Nov 02 19:47:07.096586 ip-10-0-134-13 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:47:07.475949 ip-10-0-134-13 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:47:07.475949 ip-10-0-134-13 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:47:07.475949 ip-10-0-134-13 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:47:07.475949 ip-10-0-134-13 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:47:07.479450 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479368 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.479450 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479449 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.479450 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479453 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479457 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479460 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479462 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479465 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479468 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479471 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479473 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479476 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479478 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479481 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479483 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479486 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479490 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479492 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479495 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479498 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479501 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479503 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.479540 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479506 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479514 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479517 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479519 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479522 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479526 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479530 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479533 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479536 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479538 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479541 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479544 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479546 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479549 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479552 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479555 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479558 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479561 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479564 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.479971 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479566 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479569 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479571 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479574 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479576 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479578 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479581 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479583 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479586 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479588 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479591 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479593 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479596 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479598 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479601 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479605 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479608 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479611 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479614 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479616 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.480424 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479619 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479621 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479624 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479627 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479629 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479632 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479634 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479636 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479639 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479641 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479644 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479646 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479649 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479651 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479654 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479656 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479659 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479661 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479664 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479666 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.480886 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479669 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479671 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479673 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479676 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479679 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479681 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479684 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479688 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479691 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479694 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479697 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479699 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479702 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479704 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479707 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479710 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479712 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479714 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479717 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479719 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.481383 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479722 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479724 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479728 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479730 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479735 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.479739 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480541 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480549 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480553 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480556 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480559 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480562 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480565 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480567 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480570 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480573 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480575 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480578 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480581 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.482002 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480587 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480590 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480593 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480596 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480598 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480601 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480603 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480606 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480608 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480611 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480614 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480616 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480619 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480621 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480624 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480626 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480629 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480631 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480634 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480636 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.482666 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480639 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480641 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480644 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480646 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480649 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480651 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480654 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480657 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480659 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480661 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480664 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480668 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480674 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480678 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480681 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480684 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480688 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480691 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480694 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.483150 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480696 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480699 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480701 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480704 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480707 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480709 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480711 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480714 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480717 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480720 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480722 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480725 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480727 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480730 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480732 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480735 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480737 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480740 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480743 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.483623 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480746 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480748 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480751 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480753 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480756 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480758 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480762 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480765 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480767 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480770 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480773 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480776 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480778 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480781 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480783 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480786 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480789 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480791 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480793 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480796 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.484064 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480799 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480803 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480806 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480810 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480812 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480815 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480817 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480820 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480823 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480825 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480828 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480830 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480833 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480835 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.480838 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481366 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481373 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481377 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481382 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.484535 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481385 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481389 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481391 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481395 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481398 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481400 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481403 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481405 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481408 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481410 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481413 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481415 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481418 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481421 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481423 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481426 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481428 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481431 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481433 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481435 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.485065 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481438 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481441 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481443 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481446 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481448 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481451 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481453 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481456 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481458 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481461 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481464 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481468 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481471 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481473 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481476 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481478 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481523 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481573 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.481579 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.482844 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.485565 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.482983 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.482988 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.482992 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.482995 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.482998 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483003 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483008 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483012 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483015 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483018 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483021 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483024 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483027 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483029 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483032 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483034 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483037 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483040 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483043 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.486060 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483047 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483050 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483053 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483056 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483062 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483065 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483068 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483070 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483073 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483076 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483081 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483084 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483086 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483090 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483092 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483095 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483098 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483100 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483103 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483106 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.486525 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483108 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483111 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483114 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483116 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483119 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483121 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483125 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483127 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483130 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483132 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483135 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483139 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483144 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483147 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483151 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483155 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483158 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483160 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483163 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.486993 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483166 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483168 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483171 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.483174 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484272 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484292 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484301 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484305 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484310 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484313 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484317 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484322 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484325 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484328 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484331 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484335 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484338 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484341 2579 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484344 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484347 2579 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484349 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484352 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484356 2579 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484359 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:47:07.487501 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484363 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484366 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484370 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484373 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484376 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484379 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484383 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484386 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484389 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484392 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484395 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484398 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484403 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484406 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484410 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484413 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484416 2579 flags.go:64] FLAG: --enable-server="true" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484419 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484429 2579 flags.go:64] FLAG: --event-burst="100" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484432 2579 flags.go:64] FLAG: --event-qps="50" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484435 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484438 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484441 2579 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484446 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:47:07.488060 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484449 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484452 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484455 2579 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484458 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484461 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484464 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484467 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484470 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484473 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484476 2579 flags.go:64] FLAG: --feature-gates="" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484480 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484483 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484486 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484489 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484492 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484496 2579 flags.go:64] FLAG: --help="false" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484499 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484502 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484505 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484508 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484511 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484515 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484518 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:47:07.488658 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484521 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484524 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484527 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484530 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484533 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484536 2579 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484539 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484542 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484545 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484548 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484551 2579 flags.go:64] FLAG: --lock-file="" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484554 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484557 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484560 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484566 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484568 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484572 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484576 2579 flags.go:64] FLAG: --logging-format="text" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484579 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484582 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484585 2579 flags.go:64] FLAG: --manifest-url="" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484587 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484592 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484595 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484599 2579 flags.go:64] FLAG: --max-pods="110" Nov 02 19:47:07.489192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484602 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484605 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484608 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484611 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484614 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484617 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484620 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484628 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484631 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484634 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484637 2579 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484640 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484643 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484646 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484649 2579 flags.go:64] FLAG: --port="10250" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484652 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484655 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ed9d6826d9bb117f" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484658 2579 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484661 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484664 2579 flags.go:64] FLAG: --register-node="true" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484667 2579 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484671 2579 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484673 2579 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484676 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484679 2579 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484683 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:47:07.489801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484686 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484689 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484692 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484695 2579 flags.go:64] FLAG: --runonce="false" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484698 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484701 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484704 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484707 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484709 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484712 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484715 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484720 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484724 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484727 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484730 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484732 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484736 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484739 2579 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484742 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484748 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484751 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484753 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484757 2579 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484760 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484763 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:47:07.490403 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484766 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484769 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484773 2579 flags.go:64] FLAG: --v="2" Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484780 2579 flags.go:64] FLAG: --version="false" Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484784 2579 flags.go:64] FLAG: --vmodule="" Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484788 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.484792 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484929 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484932 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484935 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484938 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484941 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484943 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484946 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484949 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484951 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484954 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484957 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484961 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484964 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484967 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.491004 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484969 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484972 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484974 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484977 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484980 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484983 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484985 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484988 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484991 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484993 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484996 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.484998 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485001 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485004 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485007 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485009 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485012 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485014 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485017 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485020 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.491571 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485023 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485025 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485028 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485030 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485033 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485036 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485039 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485042 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485044 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485048 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485051 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485054 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485056 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485059 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485063 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485067 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485070 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485074 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485077 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.492071 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485080 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485084 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485087 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485090 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485093 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485096 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485098 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485101 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485103 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485106 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485108 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485111 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485114 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485117 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485119 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485122 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485125 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485127 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485130 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.492534 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485132 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485135 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485137 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485142 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485146 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485149 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485151 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485154 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485156 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485159 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485161 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485164 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485167 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485169 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485172 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485174 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485177 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485179 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485182 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485184 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.492966 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485187 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485189 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485192 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485194 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485196 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485199 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485202 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485204 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485207 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485209 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485212 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485214 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485217 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485219 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.485246 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485296 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.493469 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485303 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485307 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485309 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485312 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485315 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485317 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485320 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485323 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485325 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485328 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485331 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485333 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485336 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485338 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485341 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485344 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485346 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485349 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485351 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.493837 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485354 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485357 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485359 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485362 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485365 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485368 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485371 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485373 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485376 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485378 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485381 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485383 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485385 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485391 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485394 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485397 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485399 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485402 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485404 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485407 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.494288 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485410 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485412 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485415 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485417 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485420 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485423 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485425 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485428 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485430 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485433 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485435 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485438 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485440 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485443 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485446 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485448 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485451 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485453 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485456 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.494767 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485459 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485463 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485465 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485469 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485473 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485476 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485482 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485486 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485488 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485492 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485496 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485499 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485502 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485504 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485507 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485510 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485512 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485515 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485518 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.495195 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485520 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485523 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485525 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485528 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485530 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485533 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485536 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485538 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485542 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485544 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485547 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485549 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485552 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485555 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485557 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485560 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485562 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485565 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485568 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485575 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.495659 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485578 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485580 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485583 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485585 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485588 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485591 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485594 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.485596 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.492728 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.492743 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.492760 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.492765 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492821 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492826 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492831 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492834 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492840 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492843 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.496112 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492845 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492848 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492851 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492853 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492857 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492861 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492864 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492867 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492870 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492873 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492875 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492878 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492883 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492886 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492888 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492891 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492893 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492896 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492899 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.496576 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492901 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492905 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492908 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492912 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492914 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492917 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492920 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492923 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492925 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492928 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492930 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492933 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492935 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492938 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492941 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492943 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492946 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492948 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492951 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.497031 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492954 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492959 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492961 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492964 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492967 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492970 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492973 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492975 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492978 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492981 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492984 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492986 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492989 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492992 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492995 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.492998 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493001 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493004 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493006 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.497513 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493009 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493012 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493014 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493017 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493019 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493022 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493024 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493027 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493029 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493032 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493034 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493037 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493039 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493042 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493044 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493047 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493049 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493052 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493054 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493057 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.497970 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493059 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493062 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493065 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493067 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493070 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493072 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493075 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493079 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493081 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493084 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493087 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493089 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493093 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493096 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493099 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493102 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493105 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493107 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493110 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.498453 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493113 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.493141 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493276 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493295 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493298 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493301 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493304 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493306 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493310 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493312 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493315 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493318 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493322 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493325 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493328 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493331 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.498902 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493334 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493337 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493339 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493343 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493346 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493348 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493351 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493354 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493356 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493359 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493361 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493364 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493366 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493369 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493371 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493374 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493376 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493379 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493381 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.499278 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493384 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493387 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493389 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493392 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493394 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493397 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493399 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493402 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493404 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493406 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493409 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493411 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493414 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493416 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493419 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493421 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493424 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493427 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493429 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493432 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.499729 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493435 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493441 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493443 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493446 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493448 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493451 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493454 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493456 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493459 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493462 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493464 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493467 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493469 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493472 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493475 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493477 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493480 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493483 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493486 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.500182 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493488 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493491 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493495 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493498 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493501 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493503 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493506 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493508 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493511 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493514 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493517 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493519 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493522 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493524 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493527 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493530 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493533 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493535 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493538 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493540 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.500678 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493543 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493545 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493548 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493550 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493553 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493555 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493558 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493560 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493563 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493566 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493568 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493571 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493573 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493575 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.493599 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493630 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:47:07.501135 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493633 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493637 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493639 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493642 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493644 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493647 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493650 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493652 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493655 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493659 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493663 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493667 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493670 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493673 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493676 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493678 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493681 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493683 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493686 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:47:07.501538 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493689 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493691 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493694 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493696 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493699 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493701 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493704 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493706 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493709 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493711 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493714 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493716 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493719 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493721 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493724 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493726 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493728 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493731 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493734 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493736 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:47:07.501974 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493739 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493741 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493744 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493746 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493749 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493751 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493754 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493757 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493759 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493762 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493764 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493767 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493769 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493771 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493774 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493776 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493779 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493782 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493784 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493786 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:47:07.502539 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493789 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493791 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493794 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493796 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493799 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493801 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493804 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493807 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493809 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493812 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493817 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493820 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493822 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493825 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493829 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493833 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493835 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493838 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493841 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:47:07.502999 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493844 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493847 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493850 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493853 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493855 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493857 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493860 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493863 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493865 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493868 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493870 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493872 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493875 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493877 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493880 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493882 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493885 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493887 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493890 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493892 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:47:07.503466 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493895 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493897 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493900 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493904 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493907 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493909 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:07.493912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.493965 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.496376 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.497432 2579 server.go:1014] "Starting client certificate rotation" Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.497526 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.498413 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.499749 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:47:07.503953 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.500901 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:47:07.512855 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.512833 2579 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:47:07.516796 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.516781 2579 log.go:25] "Validated CRI v1 image API" Nov 02 19:47:07.517637 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.517610 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:47:07.520269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.520250 2579 fs.go:135] Filesystem UUIDs: map[6a48c8a0-f570-4276-aa55-82c35a804ca1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ba7201a7-8181-4839-854e-e832b9e53533:/dev/nvme0n1p3] Nov 02 19:47:07.520346 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.520268 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 19:47:07.526377 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.526261 2579 manager.go:217] Machine: {Timestamp:2026-11-02 19:47:07.524834929 +0000 UTC m=+0.340830760 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111136 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25d5e14844b44bedf0adbba0e79914 SystemUUID:ec25d5e1-4844-b44b-edf0-adbba0e79914 BootID:7eedd5d0-d9ea-4fb9-a3fe-4d7a9e0f8e58 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:da:fc:81:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:da:fc:81:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:16:d9:91:4c:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:47:07.526377 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.526374 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:47:07.526489 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.526472 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:47:07.529415 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.529386 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:47:07.529573 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.529417 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-13.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:47:07.529650 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.529583 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:47:07.529650 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.529596 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:47:07.529731 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.529649 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:47:07.529731 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.529676 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:47:07.531015 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.531003 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:47:07.535793 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.535779 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:47:07.535927 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.535917 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:47:07.536800 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.536789 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:47:07.536862 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.536807 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:47:07.536862 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.536823 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:47:07.536862 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.536837 2579 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:47:07.536862 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.536848 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:47:07.537188 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.537175 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:47:07.539792 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.539764 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:47:07.546514 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.546501 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:47:07.547488 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547475 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:47:07.547525 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547507 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:47:07.547667 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547656 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:47:07.547702 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547675 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:47:07.547702 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547681 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:47:07.547702 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547686 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:47:07.547702 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547692 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:47:07.547702 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.547698 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:47:07.548428 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.548419 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:47:07.548465 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.548431 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:47:07.548465 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.548445 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:47:07.548465 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.548451 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:47:07.548465 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.548465 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:47:07.548579 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.548475 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:47:07.550068 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.550057 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:47:07.550109 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.550071 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:47:07.553934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.553920 2579 server.go:1263] "Started kubelet" Nov 02 19:47:07.554023 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.553999 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:47:07.554178 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.554129 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:47:07.554227 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.554202 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:47:07.554822 ip-10-0-134-13 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:47:07.555858 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.555830 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:47:07.557448 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.557434 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:47:07.562528 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.562501 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 19:47:07.565711 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.565692 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:47:07.566353 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.566338 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:47:07.566539 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.566512 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:07.567730 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.567705 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:47:07.567878 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.567858 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:47:07.567931 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.567882 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:47:07.568024 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.568013 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:47:07.568061 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.568024 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:47:07.568549 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.568527 2579 factory.go:55] Registering systemd factory Nov 02 19:47:07.568639 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.568572 2579 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:47:07.569088 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.569074 2579 factory.go:153] Registering CRI-O factory Nov 02 19:47:07.569088 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.569088 2579 factory.go:223] Registration of the crio container factory successfully Nov 02 19:47:07.569199 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.569129 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:47:07.569199 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.569148 2579 factory.go:103] Registering Raw factory Nov 02 19:47:07.569199 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.569158 2579 manager.go:1199] Started watching for new ooms in manager Nov 02 19:47:07.570240 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.570227 2579 manager.go:319] Starting recovery of all containers Nov 02 19:47:07.575023 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.574997 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:47:07.581458 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.581443 2579 manager.go:324] Recovery completed Nov 02 19:47:07.585614 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.585599 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:07.587495 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.587476 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:07.587558 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.587507 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:07.587558 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.587522 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:07.588090 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.588077 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:07.588090 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.588085 2579 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:47:07.588090 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.588093 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:47:07.588255 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.588109 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:47:07.588831 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.588808 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:07.588883 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.588853 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:07.588883 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.588871 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:07.590290 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.590270 2579 policy_none.go:50] "Start" Nov 02 19:47:07.590337 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.590298 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:47:07.590337 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.590310 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:47:07.592267 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.592255 2579 policy_none.go:44] "Start" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.637790 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.640431 2579 manager.go:341] "Starting Device Plugin manager" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.640474 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.640486 2579 server.go:87] "Starting device plugin registration server" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.640737 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.640748 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.640861 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.640952 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.640961 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.641524 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.641737 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.643005 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.643021 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.643036 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:47:07.648587 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.643069 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 19:47:07.681903 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.681832 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-13.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 19:47:07.681903 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.681860 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-13.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 19:47:07.682971 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.681847 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-13.us-west-2.compute.internal.18e4521b0225495e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-13.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-13.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:47:07.553892702 +0000 UTC m=+0.369888514,LastTimestamp:2026-11-02 19:47:07.553892702 +0000 UTC m=+0.369888514,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-13.us-west-2.compute.internal,}" Nov 02 19:47:07.740579 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.740454 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-13.us-west-2.compute.internal.18e4521b0425f9ec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-13.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-13.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-13.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:47:07.587492332 +0000 UTC m=+0.403488146,LastTimestamp:2026-11-02 19:47:07.587492332 +0000 UTC m=+0.403488146,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-13.us-west-2.compute.internal,}" Nov 02 19:47:07.740579 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.740523 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-13.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 19:47:07.740735 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.740517 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 19:47:07.740984 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.740970 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:07.741837 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.741818 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:07.741942 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.741851 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:07.741942 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.741862 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:07.741942 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.741900 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.743976 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.743955 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal"] Nov 02 19:47:07.744080 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.744019 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:07.744825 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.744805 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:07.744906 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.744833 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:07.744906 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.744848 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:07.747047 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747035 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:07.747199 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.747250 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747215 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:07.747796 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747778 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:07.747904 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747779 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:07.747904 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747829 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:07.747904 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747842 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:07.747904 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747804 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:07.747904 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.747880 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:07.751359 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.751344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.751429 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.751369 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:07.752091 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.752077 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:07.752146 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.752101 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:07.752146 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.752113 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:07.755974 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.755958 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:47:07.756037 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.756020 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pwxvw" Nov 02 19:47:07.769202 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.769180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.769269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.769208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.769269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.769224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7161e7f2a8cfc7fa3ac048d9b7921179-config\") pod \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"7161e7f2a8cfc7fa3ac048d9b7921179\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.776781 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.776762 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" node="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.779510 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.779494 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" node="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.787030 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.787008 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 19:47:07.835473 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.835448 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:07.837157 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.837141 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.837205 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.837182 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-13.us-west-2.compute.internal\": node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:07.870268 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.870244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.870360 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.870272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.870360 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.870303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7161e7f2a8cfc7fa3ac048d9b7921179-config\") pod \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"7161e7f2a8cfc7fa3ac048d9b7921179\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.870459 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.870377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.870513 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.870452 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.870561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:07.870510 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7161e7f2a8cfc7fa3ac048d9b7921179-config\") pod \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"7161e7f2a8cfc7fa3ac048d9b7921179\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:07.940486 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:07.940457 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.040785 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.040706 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.082633 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.082589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:08.084754 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.084732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:08.141183 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.141148 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.241815 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.241790 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.342751 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.342668 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.442917 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.442884 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.497518 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.497493 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 19:47:08.497975 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.497619 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 19:47:08.497975 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.497688 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a494c12af1f134ca494febab804ea946-87fdf5767f91ce37.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-13.us-west-2.compute.internal.18e4521b04267c40\": read tcp 10.0.134.13:59924->3.141.185.64:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-13.us-west-2.compute.internal.18e4521b04267c40 default 4892 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-13.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-13.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-13.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 19:47:07 +0000 UTC,LastTimestamp:2026-11-02 19:47:07.744852012 +0000 UTC m=+0.560847824,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-13.us-west-2.compute.internal,}" Nov 02 19:47:08.543391 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.543359 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.646115 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.646020 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.746302 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.746264 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.762638 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.762607 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:42:07 +0000 UTC" deadline="2028-07-26 16:05:43.944581772 +0000 UTC" Nov 02 19:47:08.762638 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.762636 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15164h18m35.181949186s" Nov 02 19:47:08.822255 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.822234 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:08.847399 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:08.847373 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:47:08.863877 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.863856 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:47:08.864998 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.864977 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:47:08.865108 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.865009 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:47:08.865108 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.865021 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:47:08.919766 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.919692 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:08.967700 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:08.967670 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:09.036719 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.036692 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:47:09.038537 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.038522 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:47:09.102051 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.102026 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:47:09.232844 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:09.232810 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d4a7ab7ffda58b80be6a631c2486f0.slice/crio-fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff WatchSource:0}: Error finding container fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff: Status 404 returned error can't find the container with id fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff Nov 02 19:47:09.238959 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.238941 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:47:09.365524 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.365498 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:09.414709 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:09.414679 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7161e7f2a8cfc7fa3ac048d9b7921179.slice/crio-5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8 WatchSource:0}: Error finding container 5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8: Status 404 returned error can't find the container with id 5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8 Nov 02 19:47:09.540359 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.540260 2579 apiserver.go:52] "Watching apiserver" Nov 02 19:47:09.566816 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.566791 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 19:47:09.650494 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.650431 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"7161e7f2a8cfc7fa3ac048d9b7921179","Type":"ContainerStarted","Data":"5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8"} Nov 02 19:47:09.651704 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.651679 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"68d4a7ab7ffda58b80be6a631c2486f0","Type":"ContainerStarted","Data":"fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff"} Nov 02 19:47:09.654301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.654256 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:47:09.654591 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.654574 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-vj7jb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-8sfkw","kube-system/global-pull-secret-syncer-j7kp4","kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal","openshift-multus/multus-5slm4","openshift-network-operator/iptables-alerter-nm6dq","kube-system/konnectivity-agent-trg79","openshift-image-registry/node-ca-pcsq4","openshift-multus/multus-additional-cni-plugins-pq2jl","openshift-network-diagnostics/network-check-target-mpxzs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js","openshift-dns/node-resolver-5bq8f","openshift-multus/network-metrics-daemon-nt26r"] Nov 02 19:47:09.658363 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.658337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.660625 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.660601 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:09.660747 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.660694 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:09.663102 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.663076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.665400 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.665383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.665496 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.665453 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:09.667616 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.667598 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:09.667695 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.667674 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:09.669876 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.669858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.672245 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.672226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.674643 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.674628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.677635 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.677614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:09.680005 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.679987 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.680799 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.680780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-systemd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.680882 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.680807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.680882 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.680825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-multus-certs\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.680882 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.680840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxznp\" (UniqueName: \"kubernetes.io/projected/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-kube-api-access-zxznp\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.680995 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.680903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-slash\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.680995 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.680941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-etc-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.680995 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.680975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681104 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-netd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681104 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-script-lib\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681104 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-bin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681216 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-kubelet\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681216 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-var-lib-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681216 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-ovn\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681314 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-log-socket\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681314 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681314 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovn-node-metrics-cert\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681314 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:09.681470 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-netns\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681470 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-netns\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681470 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-system-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681582 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-env-overrides\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681582 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.681582 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-multus\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681687 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-daemon-config\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681687 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-kubelet\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681687 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9vr\" (UniqueName: \"kubernetes.io/projected/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-kube-api-access-2z9vr\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681792 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4qd\" (UniqueName: \"kubernetes.io/projected/fea63025-526f-4fd8-b303-bc115c1226dd-kube-api-access-mf4qd\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:09.681792 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-conf-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681792 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-systemd-units\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.681896 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-kubelet-config\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.681896 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-dbus\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.681896 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.681896 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cnibin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.682026 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cni-binary-copy\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.682026 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-socket-dir-parent\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.682026 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xs6q\" (UniqueName: \"kubernetes.io/projected/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-kube-api-access-6xs6q\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.682026 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.681993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-node-log\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.682026 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-bin\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.682287 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-config\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.682287 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-k8s-cni-cncf-io\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.682287 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-hostroot\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.682287 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-etc-kubernetes\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.682287 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-iptables-alerter-script\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.682287 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-host-slash\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.682287 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:09.682655 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.682375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.684780 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.684764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.694472 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.694452 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:47:09.716347 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.716327 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7q2qk\"" Nov 02 19:47:09.716479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.716464 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:09.716535 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.716486 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:47:09.716535 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.716507 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:09.720523 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.720466 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:47:09.720655 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.720636 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:47:09.720732 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.720687 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:47:09.720732 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.720698 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:47:09.720732 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.720705 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:47:09.720868 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.720756 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:47:09.720868 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.720788 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rmffj\"" Nov 02 19:47:09.727266 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.727248 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:47:09.727416 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.727402 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:47:09.727477 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.727445 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:47:09.727539 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.727487 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fhf7l\"" Nov 02 19:47:09.727580 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.727561 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:47:09.729256 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.729240 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:47:09.729479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.729466 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:47:09.735179 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.735161 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:47:09.735262 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.735221 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p97wz\"" Nov 02 19:47:09.766856 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.766814 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7zhgs\"" Nov 02 19:47:09.766959 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.766870 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:47:09.766959 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.766870 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:47:09.766959 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.766917 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:47:09.768159 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.768135 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:47:09.768251 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.768167 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzg2s\"" Nov 02 19:47:09.768251 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.768224 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:47:09.768251 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.768240 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:47:09.768566 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.768552 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:47:09.771591 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771572 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xk7lc\"" Nov 02 19:47:09.771649 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771622 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:09.771727 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771708 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:47:09.771809 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771748 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:47:09.771809 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771760 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:47:09.771809 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771755 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 19:47:09.771809 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771762 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9wbn\"" Nov 02 19:47:09.771809 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771798 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9skmw" Nov 02 19:47:09.771992 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771818 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6t9zq\"" Nov 02 19:47:09.771992 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.771849 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:09.782939 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.782917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-agent-certs\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:09.783020 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-daemon-config\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.783058 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-kubelet\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.783092 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-cnibin\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.783092 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f40f9013-d584-4546-940b-82f31a20cd4f-host\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.783167 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-systemd-units\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.783167 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-kubelet-config\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.783167 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwfw\" (UniqueName: \"kubernetes.io/projected/207974a1-670c-49f2-b0fb-3946d261e608-kube-api-access-jkwfw\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.783275 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783196 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-kubelet\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.783275 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-kubelet-config\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.783275 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-kubernetes\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.783275 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovn-node-metrics-cert\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.783275 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783265 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-systemd-units\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.783543 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.783543 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cni-binary-copy\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.783543 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-bin\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.783543 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-daemon-config\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.783543 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783511 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-bin\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.783543 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-lib-modules\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.783798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-tuned\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.783798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-tmp\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.783798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-socket-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.783798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.783798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-iptables-alerter-script\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.783798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.783798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:09.784021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-systemd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-conf\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.784021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783895 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-systemd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f40f9013-d584-4546-940b-82f31a20cd4f-serviceca\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.784021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm8lh\" (UniqueName: \"kubernetes.io/projected/31464cb4-534f-41a2-bc06-99343f47a433-kube-api-access-tm8lh\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.784021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.783982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqs4h\" (UniqueName: \"kubernetes.io/projected/5e413f92-3561-4f46-a45b-1fbe4919af71-kube-api-access-hqs4h\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.784021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-multus-certs\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-slash\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784061 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-multus-certs\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-netd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-registration-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-device-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.784223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784221 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-slash\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784264 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-netd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-kubelet\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-ovn\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-kubelet\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-system-cni-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-ovn\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-modprobe-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784473 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-netns\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-sys\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784540 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-netns\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-host\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.784603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-system-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-env-overrides\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-system-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-systemd\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-konnectivity-ca\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-sys-fs\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.784765 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-multus\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-iptables-alerter-script\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9vr\" (UniqueName: \"kubernetes.io/projected/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-kube-api-access-2z9vr\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-multus\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.784849 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret podName:eceae17b-5b0b-45ce-87cf-9d720583c247 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:10.28482699 +0000 UTC m=+3.100822792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret") pod "global-pull-secret-syncer-j7kp4" (UID: "eceae17b-5b0b-45ce-87cf-9d720583c247") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4qd\" (UniqueName: \"kubernetes.io/projected/fea63025-526f-4fd8-b303-bc115c1226dd-kube-api-access-mf4qd\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.784967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785050 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-env-overrides\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-var-lib-kubelet\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.785412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55k45\" (UniqueName: \"kubernetes.io/projected/f40f9013-d584-4546-940b-82f31a20cd4f-kube-api-access-55k45\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31464cb4-534f-41a2-bc06-99343f47a433-tmp-dir\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-etc-selinux\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-conf-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-dbus\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpj8m\" (UniqueName: \"kubernetes.io/projected/caf2ae77-8244-44fa-a47f-1b8555fed1f2-kube-api-access-qpj8m\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785265 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-conf-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785262 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cni-binary-copy\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-dbus\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31464cb4-534f-41a2-bc06-99343f47a433-hosts-file\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.785458 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.785531 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs podName:fea63025-526f-4fd8-b303-bc115c1226dd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:10.285514715 +0000 UTC m=+3.101510531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs") pod "network-metrics-daemon-nt26r" (UID: "fea63025-526f-4fd8-b303-bc115c1226dd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cnibin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cnibin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-socket-dir-parent\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xs6q\" (UniqueName: \"kubernetes.io/projected/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-kube-api-access-6xs6q\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785635 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-socket-dir-parent\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-node-log\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-config\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785691 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-node-log\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-k8s-cni-cncf-io\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-hostroot\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-etc-kubernetes\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-host-slash\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785803 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-k8s-cni-cncf-io\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-etc-kubernetes\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-hostroot\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-script-lib\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-binary-copy\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysconfig\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.786907 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxznp\" (UniqueName: \"kubernetes.io/projected/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-kube-api-access-zxznp\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.785826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-host-slash\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-etc-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-bin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-etc-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-var-lib-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-config\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786154 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-var-lib-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786153 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-bin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-log-socket\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-run\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-log-socket\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-netns\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786314 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-netns\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.787405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.786339 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-script-lib\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.788216 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.788199 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovn-node-metrics-cert\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.844229 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.844153 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:09.844229 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.844183 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:09.844229 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.844194 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gc2hd for pod openshift-network-diagnostics/network-check-target-mpxzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:09.844420 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:09.844254 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd podName:505ade63-4db5-4709-9e6a-cdaed488b662 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:10.344237479 +0000 UTC m=+3.160233285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gc2hd" (UniqueName: "kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd") pod "network-check-target-mpxzs" (UID: "505ade63-4db5-4709-9e6a-cdaed488b662") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:09.847782 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.847757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxznp\" (UniqueName: \"kubernetes.io/projected/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-kube-api-access-zxznp\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.847900 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.847835 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xs6q\" (UniqueName: \"kubernetes.io/projected/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-kube-api-access-6xs6q\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.847900 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.847837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9vr\" (UniqueName: \"kubernetes.io/projected/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-kube-api-access-2z9vr\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.847900 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.847837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4qd\" (UniqueName: \"kubernetes.io/projected/fea63025-526f-4fd8-b303-bc115c1226dd-kube-api-access-mf4qd\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:09.886791 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-binary-copy\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.886791 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysconfig\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.886933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-run\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.886933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.886933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.886933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-agent-certs\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:09.886933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-cnibin\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.886933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f40f9013-d584-4546-940b-82f31a20cd4f-host\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.886933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886894 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysconfig\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwfw\" (UniqueName: \"kubernetes.io/projected/207974a1-670c-49f2-b0fb-3946d261e608-kube-api-access-jkwfw\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-cnibin\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-kubernetes\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-lib-modules\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887035 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-kubernetes\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.886968 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-run\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-tuned\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887064 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f40f9013-d584-4546-940b-82f31a20cd4f-host\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-tmp\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-socket-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887203 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-lib-modules\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-conf\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f40f9013-d584-4546-940b-82f31a20cd4f-serviceca\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8lh\" (UniqueName: \"kubernetes.io/projected/31464cb4-534f-41a2-bc06-99343f47a433-kube-api-access-tm8lh\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.887301 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqs4h\" (UniqueName: \"kubernetes.io/projected/5e413f92-3561-4f46-a45b-1fbe4919af71-kube-api-access-hqs4h\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-socket-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-registration-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887430 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-registration-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-device-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-system-cni-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-modprobe-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887540 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-binary-copy\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-sys\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-sys\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887602 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-host\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-host\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-device-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-systemd\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-konnectivity-ca\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887729 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-system-cni-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.888174 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-sys-fs\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887791 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-systemd\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-var-lib-kubelet\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55k45\" (UniqueName: \"kubernetes.io/projected/f40f9013-d584-4546-940b-82f31a20cd4f-kube-api-access-55k45\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-var-lib-kubelet\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31464cb4-534f-41a2-bc06-99343f47a433-tmp-dir\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-etc-selinux\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj8m\" (UniqueName: \"kubernetes.io/projected/caf2ae77-8244-44fa-a47f-1b8555fed1f2-kube-api-access-qpj8m\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.887973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31464cb4-534f-41a2-bc06-99343f47a433-hosts-file\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888136 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-modprobe-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888290 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-conf\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888345 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-sys-fs\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31464cb4-534f-41a2-bc06-99343f47a433-hosts-file\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-etc-selinux\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888534 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f40f9013-d584-4546-940b-82f31a20cd4f-serviceca\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.889064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888549 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.889712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888939 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.889712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.888976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31464cb4-534f-41a2-bc06-99343f47a433-tmp-dir\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.889712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.889215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-konnectivity-ca\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:09.889712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.889466 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-tmp\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.889712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.889582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-agent-certs\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:09.889712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.889663 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-tuned\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.950003 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.949968 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8lh\" (UniqueName: \"kubernetes.io/projected/31464cb4-534f-41a2-bc06-99343f47a433-kube-api-access-tm8lh\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.950225 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.950194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwfw\" (UniqueName: \"kubernetes.io/projected/207974a1-670c-49f2-b0fb-3946d261e608-kube-api-access-jkwfw\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:09.950308 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.950230 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqs4h\" (UniqueName: \"kubernetes.io/projected/5e413f92-3561-4f46-a45b-1fbe4919af71-kube-api-access-hqs4h\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:09.950308 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.950196 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj8m\" (UniqueName: \"kubernetes.io/projected/caf2ae77-8244-44fa-a47f-1b8555fed1f2-kube-api-access-qpj8m\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:09.950945 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.950925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55k45\" (UniqueName: \"kubernetes.io/projected/f40f9013-d584-4546-940b-82f31a20cd4f-kube-api-access-55k45\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.971012 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.970991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:47:09.981474 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:09.981450 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184c44a1_6433_4fb3_a7cd_ad9fa39cb642.slice/crio-d9b5284b4e60cf6be7d5f6e777159aee58564a7ca64537817b1c281a15ca0ec0 WatchSource:0}: Error finding container d9b5284b4e60cf6be7d5f6e777159aee58564a7ca64537817b1c281a15ca0ec0: Status 404 returned error can't find the container with id d9b5284b4e60cf6be7d5f6e777159aee58564a7ca64537817b1c281a15ca0ec0 Nov 02 19:47:09.988377 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.988351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5slm4" Nov 02 19:47:09.992406 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.992386 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:47:09.997408 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.997387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:09.999438 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:09.999422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:47:09.999998 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:09.999972 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbc31d0_a7a3_4898_8841_9ed8ebf5ff21.slice/crio-b994a73ee0369bbc0bb09f9675ed8f4ce063a5edea9cc12532d9e271df27a7fb WatchSource:0}: Error finding container b994a73ee0369bbc0bb09f9675ed8f4ce063a5edea9cc12532d9e271df27a7fb: Status 404 returned error can't find the container with id b994a73ee0369bbc0bb09f9675ed8f4ce063a5edea9cc12532d9e271df27a7fb Nov 02 19:47:10.001679 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.001659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:10.003212 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:10.003189 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40f9013_d584_4546_940b_82f31a20cd4f.slice/crio-f4c0f792a4995f4c1b4cd4406f9b962ce6a321c5653108719d490513ef7a8a79 WatchSource:0}: Error finding container f4c0f792a4995f4c1b4cd4406f9b962ce6a321c5653108719d490513ef7a8a79: Status 404 returned error can't find the container with id f4c0f792a4995f4c1b4cd4406f9b962ce6a321c5653108719d490513ef7a8a79 Nov 02 19:47:10.009696 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:10.009665 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77546eb8_8ef9_46b1_a47a_ca80ca5f76ee.slice/crio-e4fac70cb6dc09857b70c7ff6bdf04a10c33ddd3311ada3153e3915f38c58041 WatchSource:0}: Error finding container e4fac70cb6dc09857b70c7ff6bdf04a10c33ddd3311ada3153e3915f38c58041: Status 404 returned error can't find the container with id e4fac70cb6dc09857b70c7ff6bdf04a10c33ddd3311ada3153e3915f38c58041 Nov 02 19:47:10.012600 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:10.012576 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31464cb4_534f_41a2_bc06_99343f47a433.slice/crio-e06b5369cade018e3a47d004cbdd99da6ecf45fa7e05f4a4323a93565d574863 WatchSource:0}: Error finding container e06b5369cade018e3a47d004cbdd99da6ecf45fa7e05f4a4323a93565d574863: Status 404 returned error can't find the container with id e06b5369cade018e3a47d004cbdd99da6ecf45fa7e05f4a4323a93565d574863 Nov 02 19:47:10.015596 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:10.015580 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9766ef_0c1a_4ddc_9918_e0c1fb68aada.slice/crio-bca6f05f35d6071ff8afd2ce520c1438e201ac036ed294b5b08564f5dda2f145 WatchSource:0}: Error finding container bca6f05f35d6071ff8afd2ce520c1438e201ac036ed294b5b08564f5dda2f145: Status 404 returned error can't find the container with id bca6f05f35d6071ff8afd2ce520c1438e201ac036ed294b5b08564f5dda2f145 Nov 02 19:47:10.023181 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.023151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:47:10.027202 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.027183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:47:10.032120 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.032105 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:47:10.034582 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:10.034559 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf2ae77_8244_44fa_a47f_1b8555fed1f2.slice/crio-288a2b09d1a183183f03dcb7e27d2f4887c1ca6a6375df990cac8085e7d67b01 WatchSource:0}: Error finding container 288a2b09d1a183183f03dcb7e27d2f4887c1ca6a6375df990cac8085e7d67b01: Status 404 returned error can't find the container with id 288a2b09d1a183183f03dcb7e27d2f4887c1ca6a6375df990cac8085e7d67b01 Nov 02 19:47:10.039247 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:10.039219 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207974a1_670c_49f2_b0fb_3946d261e608.slice/crio-2d1b024470a51ec41f4cd244ffeb0f6d1665ed19789513a78e31ee72f2c08eb2 WatchSource:0}: Error finding container 2d1b024470a51ec41f4cd244ffeb0f6d1665ed19789513a78e31ee72f2c08eb2: Status 404 returned error can't find the container with id 2d1b024470a51ec41f4cd244ffeb0f6d1665ed19789513a78e31ee72f2c08eb2 Nov 02 19:47:10.042096 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:10.042072 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e413f92_3561_4f46_a45b_1fbe4919af71.slice/crio-f9f46b44620d281d218226a68c8fc2fbe6a90ea6665c3178f4c214555053193e WatchSource:0}: Error finding container f9f46b44620d281d218226a68c8fc2fbe6a90ea6665c3178f4c214555053193e: Status 404 returned error can't find the container with id f9f46b44620d281d218226a68c8fc2fbe6a90ea6665c3178f4c214555053193e Nov 02 19:47:10.292319 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.292220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:10.292319 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.292280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:10.292503 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.292386 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:10.292503 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.292402 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:10.292503 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.292449 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret podName:eceae17b-5b0b-45ce-87cf-9d720583c247 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:11.292432953 +0000 UTC m=+4.108428756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret") pod "global-pull-secret-syncer-j7kp4" (UID: "eceae17b-5b0b-45ce-87cf-9d720583c247") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:10.292503 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.292468 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs podName:fea63025-526f-4fd8-b303-bc115c1226dd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:11.292456954 +0000 UTC m=+4.108452756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs") pod "network-metrics-daemon-nt26r" (UID: "fea63025-526f-4fd8-b303-bc115c1226dd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:10.393669 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.393629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:10.393832 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.393784 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:10.393832 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.393806 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:10.393832 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.393816 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gc2hd for pod openshift-network-diagnostics/network-check-target-mpxzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:10.394073 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:10.393887 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd podName:505ade63-4db5-4709-9e6a-cdaed488b662 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:11.393870537 +0000 UTC m=+4.209866354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc2hd" (UniqueName: "kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd") pod "network-check-target-mpxzs" (UID: "505ade63-4db5-4709-9e6a-cdaed488b662") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:10.654711 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.654626 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" event={"ID":"5e413f92-3561-4f46-a45b-1fbe4919af71","Type":"ContainerStarted","Data":"f9f46b44620d281d218226a68c8fc2fbe6a90ea6665c3178f4c214555053193e"} Nov 02 19:47:10.655868 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.655830 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerStarted","Data":"2d1b024470a51ec41f4cd244ffeb0f6d1665ed19789513a78e31ee72f2c08eb2"} Nov 02 19:47:10.657085 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.657055 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" event={"ID":"caf2ae77-8244-44fa-a47f-1b8555fed1f2","Type":"ContainerStarted","Data":"288a2b09d1a183183f03dcb7e27d2f4887c1ca6a6375df990cac8085e7d67b01"} Nov 02 19:47:10.658118 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.658088 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-trg79" event={"ID":"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada","Type":"ContainerStarted","Data":"bca6f05f35d6071ff8afd2ce520c1438e201ac036ed294b5b08564f5dda2f145"} Nov 02 19:47:10.659216 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.659194 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5bq8f" event={"ID":"31464cb4-534f-41a2-bc06-99343f47a433","Type":"ContainerStarted","Data":"e06b5369cade018e3a47d004cbdd99da6ecf45fa7e05f4a4323a93565d574863"} Nov 02 19:47:10.660284 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.660263 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5slm4" event={"ID":"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21","Type":"ContainerStarted","Data":"b994a73ee0369bbc0bb09f9675ed8f4ce063a5edea9cc12532d9e271df27a7fb"} Nov 02 19:47:10.661407 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.661385 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"e4fac70cb6dc09857b70c7ff6bdf04a10c33ddd3311ada3153e3915f38c58041"} Nov 02 19:47:10.662825 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.662803 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pcsq4" event={"ID":"f40f9013-d584-4546-940b-82f31a20cd4f","Type":"ContainerStarted","Data":"f4c0f792a4995f4c1b4cd4406f9b962ce6a321c5653108719d490513ef7a8a79"} Nov 02 19:47:10.664173 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.664153 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nm6dq" event={"ID":"184c44a1-6433-4fb3-a7cd-ad9fa39cb642","Type":"ContainerStarted","Data":"d9b5284b4e60cf6be7d5f6e777159aee58564a7ca64537817b1c281a15ca0ec0"} Nov 02 19:47:10.775379 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.775306 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:09 +0000 UTC" deadline="2028-04-08 04:51:39.548178082 +0000 UTC" Nov 02 19:47:10.775379 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:10.775376 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12537h4m28.772807597s" Nov 02 19:47:11.302255 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.301822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:11.302255 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.301879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:11.302255 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.302055 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:11.302255 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.302115 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs podName:fea63025-526f-4fd8-b303-bc115c1226dd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:13.302098041 +0000 UTC m=+6.118093858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs") pod "network-metrics-daemon-nt26r" (UID: "fea63025-526f-4fd8-b303-bc115c1226dd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:11.302255 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.302180 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:11.302255 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.302221 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret podName:eceae17b-5b0b-45ce-87cf-9d720583c247 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:13.302209664 +0000 UTC m=+6.118205483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret") pod "global-pull-secret-syncer-j7kp4" (UID: "eceae17b-5b0b-45ce-87cf-9d720583c247") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:11.403018 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.402976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:11.403215 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.403163 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:11.403215 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.403186 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:11.403215 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.403199 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gc2hd for pod openshift-network-diagnostics/network-check-target-mpxzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:11.403514 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.403263 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd podName:505ade63-4db5-4709-9e6a-cdaed488b662 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:13.403243871 +0000 UTC m=+6.219239674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc2hd" (UniqueName: "kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd") pod "network-check-target-mpxzs" (UID: "505ade63-4db5-4709-9e6a-cdaed488b662") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:11.646699 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.646622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:11.646862 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.646773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:11.646931 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.646908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:11.647027 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.647006 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:11.647459 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.647438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:11.647570 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:11.647541 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:11.775986 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.775939 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:09 +0000 UTC" deadline="2028-03-30 08:48:39.744575508 +0000 UTC" Nov 02 19:47:11.775986 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:11.775984 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12325h1m27.968597191s" Nov 02 19:47:13.318361 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.318155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:13.318909 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.318388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:13.318909 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.318495 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:13.318909 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.318557 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs podName:fea63025-526f-4fd8-b303-bc115c1226dd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:17.318539573 +0000 UTC m=+10.134535394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs") pod "network-metrics-daemon-nt26r" (UID: "fea63025-526f-4fd8-b303-bc115c1226dd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:13.318909 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.318620 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:13.318909 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.318653 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret podName:eceae17b-5b0b-45ce-87cf-9d720583c247 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:17.318641527 +0000 UTC m=+10.134637343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret") pod "global-pull-secret-syncer-j7kp4" (UID: "eceae17b-5b0b-45ce-87cf-9d720583c247") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:13.418921 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.418890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:13.419129 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.419113 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:13.419195 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.419136 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:13.419195 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.419148 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gc2hd for pod openshift-network-diagnostics/network-check-target-mpxzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:13.419309 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.419232 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd podName:505ade63-4db5-4709-9e6a-cdaed488b662 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:17.419214131 +0000 UTC m=+10.235209949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc2hd" (UniqueName: "kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd") pod "network-check-target-mpxzs" (UID: "505ade63-4db5-4709-9e6a-cdaed488b662") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:13.646370 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.645714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:13.646370 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.645863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:13.646370 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.645952 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:13.646370 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.646069 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:13.646370 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.646136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:13.646370 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:13.646216 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:13.694257 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.694038 2579 generic.go:356] "Generic (PLEG): container finished" podID="68d4a7ab7ffda58b80be6a631c2486f0" containerID="83ec6f649c39f602c775a78b3b77b499c34977895f5d6309a0b978b4a9827d53" exitCode=0 Nov 02 19:47:13.694257 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.694184 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"68d4a7ab7ffda58b80be6a631c2486f0","Type":"ContainerDied","Data":"83ec6f649c39f602c775a78b3b77b499c34977895f5d6309a0b978b4a9827d53"} Nov 02 19:47:13.703838 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:13.703810 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerStarted","Data":"c8f73ca55198688762ef07593815697c8675a62c972af43f4e709adb9b7820f3"} Nov 02 19:47:14.714017 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:14.713375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"7161e7f2a8cfc7fa3ac048d9b7921179","Type":"ContainerStarted","Data":"5eb6979ad7a74feec2c4f8db83e9fcfc1cf1a4f79d03dd5a07ef379ba99faf2e"} Nov 02 19:47:14.719309 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:14.719204 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"68d4a7ab7ffda58b80be6a631c2486f0","Type":"ContainerStarted","Data":"42c3576ba5fee9e633296f5a759ede3371e925f72b0773b067ab8b671131947e"} Nov 02 19:47:14.721705 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:14.721666 2579 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="c8f73ca55198688762ef07593815697c8675a62c972af43f4e709adb9b7820f3" exitCode=0 Nov 02 19:47:14.721862 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:14.721748 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerDied","Data":"c8f73ca55198688762ef07593815697c8675a62c972af43f4e709adb9b7820f3"} Nov 02 19:47:14.736116 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:14.736083 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-trg79" event={"ID":"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada","Type":"ContainerStarted","Data":"b14ba8063f3e237b62cee0db1c0457b92b2db4e3375711083de0583f500c2904"} Nov 02 19:47:14.825031 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:14.824958 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:14.831524 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:14.831454 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" podStartSLOduration=5.831436018 podStartE2EDuration="5.831436018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:14.830358606 +0000 UTC m=+7.646354428" watchObservedRunningTime="2026-11-02 19:47:14.831436018 +0000 UTC m=+7.647431841" Nov 02 19:47:15.068261 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:15.068203 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-trg79" podStartSLOduration=2.561579874 podStartE2EDuration="7.068186165s" totalImagesPullingTime="4.506606291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:14.946232608 +0000 UTC m=+7.762228424" watchObservedRunningTime="2026-11-02 19:47:15.068186165 +0000 UTC m=+7.884181995" Nov 02 19:47:15.643847 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:15.643818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:15.644007 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:15.643934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:15.644070 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:15.644005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:15.644124 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:15.644085 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:15.644175 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:15.644138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:15.644237 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:15.644217 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:16.746308 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:16.745990 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pcsq4" event={"ID":"f40f9013-d584-4546-940b-82f31a20cd4f","Type":"ContainerStarted","Data":"ac28ae4d2949a984a8b33a45d89dda6bf9df30ec94b329099f4173e65a8e444a"} Nov 02 19:47:16.749236 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:16.749204 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" event={"ID":"5e413f92-3561-4f46-a45b-1fbe4919af71","Type":"ContainerStarted","Data":"1c6e8b3e5a9b68dcbe612f3ca2ac94b9f291a2ec9b676e1d4d549a775851732e"} Nov 02 19:47:16.862369 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:16.861364 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pcsq4" podStartSLOduration=3.296042325 podStartE2EDuration="8.861345686s" totalImagesPullingTime="5.565303361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:16.861211043 +0000 UTC m=+9.677206867" watchObservedRunningTime="2026-11-02 19:47:16.861345686 +0000 UTC m=+9.677341509" Nov 02 19:47:16.862369 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:16.861874 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" podStartSLOduration=7.861860156 podStartE2EDuration="7.861860156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:15.184482539 +0000 UTC m=+8.000478375" watchObservedRunningTime="2026-11-02 19:47:16.861860156 +0000 UTC m=+9.677855979" Nov 02 19:47:17.353905 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.353864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:17.354067 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.353927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:17.354106 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.354075 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:17.354157 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.354145 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs podName:fea63025-526f-4fd8-b303-bc115c1226dd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:25.354124727 +0000 UTC m=+18.170120591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs") pod "network-metrics-daemon-nt26r" (UID: "fea63025-526f-4fd8-b303-bc115c1226dd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:17.354237 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.354223 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:17.354320 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.354267 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret podName:eceae17b-5b0b-45ce-87cf-9d720583c247 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:25.354254768 +0000 UTC m=+18.170250605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret") pod "global-pull-secret-syncer-j7kp4" (UID: "eceae17b-5b0b-45ce-87cf-9d720583c247") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:17.454780 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.454744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:17.454953 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.454912 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:17.454953 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.454927 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:17.454953 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.454937 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gc2hd for pod openshift-network-diagnostics/network-check-target-mpxzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:17.455103 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.454982 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd podName:505ade63-4db5-4709-9e6a-cdaed488b662 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:25.454970882 +0000 UTC m=+18.270966681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc2hd" (UniqueName: "kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd") pod "network-check-target-mpxzs" (UID: "505ade63-4db5-4709-9e6a-cdaed488b662") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:17.646599 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.646540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:17.646726 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.646628 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:17.646726 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.646669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:17.646726 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.646711 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:17.646824 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.646739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:17.646824 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:17.646788 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:17.937063 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.936583 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:17.937532 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:17.937201 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:18.238224 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.238166 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:47:18.650401 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.650242 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:47:18.238190873Z","UUID":"169ea9c7-4e1d-42f2-a281-8f03ca46f7c8","Handler":null,"Name":"","Endpoint":""} Nov 02 19:47:18.652153 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.652132 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:47:18.652300 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.652161 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:47:18.755223 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.755185 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" event={"ID":"5e413f92-3561-4f46-a45b-1fbe4919af71","Type":"ContainerStarted","Data":"56b4c5774a87a63b8bfc14a963f6dc629dd2f1f90e81d6030ed347cb2257bfe6"} Nov 02 19:47:18.756787 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.756758 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5bq8f" event={"ID":"31464cb4-534f-41a2-bc06-99343f47a433","Type":"ContainerStarted","Data":"b770f0cde37c19f8c5b1d4dbfea87f587c60fc4f972b90f1ba54fce05455bdc5"} Nov 02 19:47:18.758384 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.758360 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5slm4" event={"ID":"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21","Type":"ContainerStarted","Data":"24846d6b32503e26b0a8097fc7d093eeb08e059cb3ddea1e4b145f4e2adc0d94"} Nov 02 19:47:18.758915 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.758894 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-trg79" Nov 02 19:47:18.871743 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:18.871688 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5bq8f" podStartSLOduration=2.837735571 podStartE2EDuration="10.871671598s" totalImagesPullingTime="8.033936027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:18.87097094 +0000 UTC m=+11.686966779" watchObservedRunningTime="2026-11-02 19:47:18.871671598 +0000 UTC m=+11.687667421" Nov 02 19:47:19.108807 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:19.108755 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5slm4" podStartSLOduration=3.059458361 podStartE2EDuration="11.108736633s" totalImagesPullingTime="8.049278272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:19.107985704 +0000 UTC m=+11.923981528" watchObservedRunningTime="2026-11-02 19:47:19.108736633 +0000 UTC m=+11.924732482" Nov 02 19:47:19.643728 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:19.643699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:19.643728 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:19.643712 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:19.643961 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:19.643873 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:19.643961 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:19.643908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:19.644038 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:19.643962 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:19.644038 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:19.644018 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:20.766266 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:20.765797 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" event={"ID":"5e413f92-3561-4f46-a45b-1fbe4919af71","Type":"ContainerStarted","Data":"5d2f602169003ece391daf278dfde5e11b43df8362f673b98deb49e6d127de74"} Nov 02 19:47:20.767687 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:20.767656 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" event={"ID":"caf2ae77-8244-44fa-a47f-1b8555fed1f2","Type":"ContainerStarted","Data":"6bfbe1b19b9565342ebf873b62bd4f16a7c86c5fd3f14ea002cbb6fee98e2fa7"} Nov 02 19:47:20.769138 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:20.769112 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nm6dq" event={"ID":"184c44a1-6433-4fb3-a7cd-ad9fa39cb642","Type":"ContainerStarted","Data":"7aee27f704c87c8d5b24293c7eda4283457118ac7480d73bda88eabd8c6d9403"} Nov 02 19:47:20.882478 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:20.882408 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" podStartSLOduration=3.503068751 podStartE2EDuration="12.882388696s" totalImagesPullingTime="9.379319945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:20.881940607 +0000 UTC m=+13.697936448" watchObservedRunningTime="2026-11-02 19:47:20.882388696 +0000 UTC m=+13.698384535" Nov 02 19:47:20.996970 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:20.996923 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nm6dq" podStartSLOduration=4.715241257 podStartE2EDuration="12.996908828s" totalImagesPullingTime="8.281667571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:20.996804457 +0000 UTC m=+13.812800280" watchObservedRunningTime="2026-11-02 19:47:20.996908828 +0000 UTC m=+13.812904649" Nov 02 19:47:21.112492 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:21.112402 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" podStartSLOduration=2.8846756080000002 podStartE2EDuration="13.112386835s" totalImagesPullingTime="10.227711227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:21.112171717 +0000 UTC m=+13.928167538" watchObservedRunningTime="2026-11-02 19:47:21.112386835 +0000 UTC m=+13.928382657" Nov 02 19:47:21.643845 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:21.643803 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:21.644030 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:21.643938 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:21.644030 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:21.643953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:21.644125 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:21.643942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:21.644125 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:21.644054 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:21.644191 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:21.644124 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:21.773320 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:21.773266 2579 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="c6882e11c0ca8ca453556a16e765d91f344c167f124b056a3919b6ba86a51fc7" exitCode=0 Nov 02 19:47:21.774020 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:21.773345 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerDied","Data":"c6882e11c0ca8ca453556a16e765d91f344c167f124b056a3919b6ba86a51fc7"} Nov 02 19:47:23.644019 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:23.643813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:23.644738 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:23.643813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:23.644738 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:23.644108 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:23.644738 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:23.644169 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:23.644738 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:23.643879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:23.644738 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:23.644261 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:23.779563 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:23.779529 2579 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="5eae745bd2f0c80b122a486a8e39821e45fca1c7aea2f4b36eba67cff5c13234" exitCode=0 Nov 02 19:47:23.779714 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:23.779575 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerDied","Data":"5eae745bd2f0c80b122a486a8e39821e45fca1c7aea2f4b36eba67cff5c13234"} Nov 02 19:47:25.412694 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.412661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:25.413083 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.412708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:25.413083 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.412833 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:25.413083 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.412909 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs podName:fea63025-526f-4fd8-b303-bc115c1226dd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:41.412891268 +0000 UTC m=+34.228887089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs") pod "network-metrics-daemon-nt26r" (UID: "fea63025-526f-4fd8-b303-bc115c1226dd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:25.413083 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.412834 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:25.413083 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.412959 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret podName:eceae17b-5b0b-45ce-87cf-9d720583c247 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:41.412949159 +0000 UTC m=+34.228944973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret") pod "global-pull-secret-syncer-j7kp4" (UID: "eceae17b-5b0b-45ce-87cf-9d720583c247") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:25.513846 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.513807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:25.514062 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.513947 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:25.514062 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.513969 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:25.514062 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.513980 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gc2hd for pod openshift-network-diagnostics/network-check-target-mpxzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:25.514062 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.514040 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd podName:505ade63-4db5-4709-9e6a-cdaed488b662 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:41.514022337 +0000 UTC m=+34.330018151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc2hd" (UniqueName: "kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd") pod "network-check-target-mpxzs" (UID: "505ade63-4db5-4709-9e6a-cdaed488b662") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:25.643307 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.643222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:25.643307 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.643243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:25.643506 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.643344 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:25.643506 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.643403 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:25.643506 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.643438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:25.643603 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:25.643535 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:25.786495 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.786452 2579 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="5616231cbaa0c19d40414acc403b2fae3bf6f87ce7eb7a2d5602cfe65358301c" exitCode=0 Nov 02 19:47:25.786666 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:25.786513 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerDied","Data":"5616231cbaa0c19d40414acc403b2fae3bf6f87ce7eb7a2d5602cfe65358301c"} Nov 02 19:47:27.645424 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:27.645393 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:27.645894 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:27.645502 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:27.645894 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:27.645860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:27.645991 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:27.645936 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:27.646244 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:27.646220 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:27.646377 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:27.646356 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:29.644004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:29.643864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:29.644004 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:29.643911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:29.644556 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:29.644008 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:29.644556 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:29.644320 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:29.644556 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:29.644390 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:29.644556 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:29.644459 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:29.798351 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:29.798319 2579 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="1786d35d12092f97eafaad66da7959b09c246b0fbd3164893951215157b93c8c" exitCode=0 Nov 02 19:47:29.798510 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:29.798373 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerDied","Data":"1786d35d12092f97eafaad66da7959b09c246b0fbd3164893951215157b93c8c"} Nov 02 19:47:30.804194 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.804010 2579 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="c5bacd6bdee74d38592f0e1f2a8a51a243236096cfc6af0f4a67f86af51a9bfe" exitCode=0 Nov 02 19:47:30.804670 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.804093 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerDied","Data":"c5bacd6bdee74d38592f0e1f2a8a51a243236096cfc6af0f4a67f86af51a9bfe"} Nov 02 19:47:30.807295 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.807262 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"5b2074e7231b73644189506ac00778c30818d4709bddab63f8aa50e87fa9680f"} Nov 02 19:47:30.807380 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.807332 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"7529ae094987b2d820f45edc668c9c67d3f06d29489a8aa633f852b6398c1c68"} Nov 02 19:47:30.807380 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.807363 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"9a681482bd242cedacd6f0163bed15afb5c574b8294a2c9edaa78666da90132b"} Nov 02 19:47:30.807437 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.807388 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"5e95a336c95d85be366f40811fcb16f8c47bdd972a7a3b81a566f77cbea9606a"} Nov 02 19:47:30.807437 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.807418 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"f95eeb802c4a538cdf42fb9c6a41887bdd25a4018e6f59674de792b6f2cc701c"} Nov 02 19:47:30.807494 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:30.807443 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"aafc5e1c198f205f3bced6f65927716cf3390413098dcd4b4010423e20e20b15"} Nov 02 19:47:31.643860 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:31.643829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:31.643860 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:31.643849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:31.644207 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:31.643954 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:31.644207 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:31.643998 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:31.644207 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:31.644050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:31.644207 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:31.644136 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:31.812548 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:31.812519 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" event={"ID":"207974a1-670c-49f2-b0fb-3946d261e608","Type":"ContainerStarted","Data":"04fed35786c7fba3fc5ba776fba9d5b0d401ce6d2b86f37c7692b71de069d41b"} Nov 02 19:47:31.932445 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:31.932349 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pq2jl" podStartSLOduration=8.318510783 podStartE2EDuration="23.932333012s" totalImagesPullingTime="15.613822229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:31.931833206 +0000 UTC m=+24.747829028" watchObservedRunningTime="2026-11-02 19:47:31.932333012 +0000 UTC m=+24.748328834" Nov 02 19:47:32.818077 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:32.817991 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"2b7b20e72354ee11864f43a19cc18e0d9cf87f0ec6f9675d62cb28db78e2b7e7"} Nov 02 19:47:33.643443 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:33.643413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:33.643589 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:33.643422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:33.643589 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:33.643508 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:33.643589 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:33.643427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:33.643589 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:33.643578 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:33.643727 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:33.643684 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:34.827937 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:34.827847 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" event={"ID":"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee","Type":"ContainerStarted","Data":"3d85cf48aee97a3d47d08169b6888a0a8f720db3f5edd542a28f54a569b1b999"} Nov 02 19:47:34.828719 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:34.828179 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:34.842958 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:34.842934 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:35.103199 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:35.103104 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" podStartSLOduration=7.024992827 podStartE2EDuration="27.103090608s" totalImagesPullingTime="20.078097781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:35.102616343 +0000 UTC m=+27.918612167" watchObservedRunningTime="2026-11-02 19:47:35.103090608 +0000 UTC m=+27.919086429" Nov 02 19:47:35.644172 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:35.644135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:35.644172 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:35.644167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:35.644401 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:35.644331 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:35.644401 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:35.644365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:35.644471 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:35.644443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:35.644531 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:35.644516 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:35.840214 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:35.840129 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:35.840780 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:35.840764 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:35.858020 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:35.857994 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:47:37.182257 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:37.182225 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j7kp4"] Nov 02 19:47:37.182642 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:37.182345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:37.182642 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:37.182440 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:37.182765 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:37.182742 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mpxzs"] Nov 02 19:47:37.182835 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:37.182824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:37.182916 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:37.182900 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:37.232919 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:37.232886 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nt26r"] Nov 02 19:47:37.233054 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:37.232980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:37.233087 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:37.233059 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:38.643841 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:38.643809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:38.643841 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:38.643839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:38.644200 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:38.644009 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:38.644200 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:38.644057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:38.644200 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:38.644150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:38.644351 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:38.644243 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:40.643206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:40.643178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:40.643646 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:40.643179 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:40.643646 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:40.643304 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:40.643646 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:40.643179 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:40.643646 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:40.643481 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:40.643646 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:40.643407 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:41.430758 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:41.430718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:41.430758 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:41.430764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:41.431054 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.430879 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:41.431054 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.430938 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs podName:fea63025-526f-4fd8-b303-bc115c1226dd nodeName:}" failed. No retries permitted until 2026-11-02 19:48:13.43092483 +0000 UTC m=+66.246920643 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs") pod "network-metrics-daemon-nt26r" (UID: "fea63025-526f-4fd8-b303-bc115c1226dd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 19:47:41.431054 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.430880 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:41.431054 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.430973 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret podName:eceae17b-5b0b-45ce-87cf-9d720583c247 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:13.430963814 +0000 UTC m=+66.246959627 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret") pod "global-pull-secret-syncer-j7kp4" (UID: "eceae17b-5b0b-45ce-87cf-9d720583c247") : object "kube-system"/"original-pull-secret" not registered Nov 02 19:47:41.531654 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:41.531618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:41.531823 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.531753 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 19:47:41.531823 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.531767 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 19:47:41.531823 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.531777 2579 projected.go:196] Error preparing data for projected volume kube-api-access-gc2hd for pod openshift-network-diagnostics/network-check-target-mpxzs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:41.531823 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:41.531821 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd podName:505ade63-4db5-4709-9e6a-cdaed488b662 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:13.531810356 +0000 UTC m=+66.347806156 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gc2hd" (UniqueName: "kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd") pod "network-check-target-mpxzs" (UID: "505ade63-4db5-4709-9e6a-cdaed488b662") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 19:47:42.643405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:42.643374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:42.643790 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:42.643417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:42.643790 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:42.643375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:42.643790 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:42.643480 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpxzs" podUID="505ade63-4db5-4709-9e6a-cdaed488b662" Nov 02 19:47:42.643790 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:42.643557 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7kp4" podUID="eceae17b-5b0b-45ce-87cf-9d720583c247" Nov 02 19:47:42.643790 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:42.643622 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nt26r" podUID="fea63025-526f-4fd8-b303-bc115c1226dd" Nov 02 19:47:42.963704 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:42.963635 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeReady" Nov 02 19:47:42.963832 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:42.963765 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 19:47:43.085569 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.085536 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-t25fz"] Nov 02 19:47:43.132866 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.132840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-t25fz" Nov 02 19:47:43.134838 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.134801 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qwntm"] Nov 02 19:47:43.146920 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.146899 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9"] Nov 02 19:47:43.147021 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.147004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qwntm" Nov 02 19:47:43.159648 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.159624 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6s4x6"] Nov 02 19:47:43.159874 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.159788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" Nov 02 19:47:43.180102 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.180076 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7"] Nov 02 19:47:43.180226 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.180203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" Nov 02 19:47:43.195980 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.195951 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc"] Nov 02 19:47:43.196585 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.196321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.218597 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.218534 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kg2ng"] Nov 02 19:47:43.218723 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.218650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.233001 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.232973 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2b245"] Nov 02 19:47:43.233126 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.233086 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.238603 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.238579 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.239402 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.239121 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.243574 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.243546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggbz\" (UniqueName: \"kubernetes.io/projected/f98defbd-aa11-4a76-bac3-4d7aacdb44a1-kube-api-access-7ggbz\") pod \"kas-connection-checker-685895c7c9-t25fz\" (UID: \"f98defbd-aa11-4a76-bac3-4d7aacdb44a1\") " pod="kube-system/kas-connection-checker-685895c7c9-t25fz" Nov 02 19:47:43.243680 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.243588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mcnh\" (UniqueName: \"kubernetes.io/projected/cb5fc542-006a-4da5-b985-cc7b0f1709a7-kube-api-access-8mcnh\") pod \"kas-connection-checker-685895c7c9-qwntm\" (UID: \"cb5fc542-006a-4da5-b985-cc7b0f1709a7\") " pod="kube-system/kas-connection-checker-685895c7c9-qwntm" Nov 02 19:47:43.243680 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.243633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9m9j\" (UniqueName: \"kubernetes.io/projected/e5640e81-834c-4854-9cc9-589ccbd39d64-kube-api-access-s9m9j\") pod \"kas-connection-checker-685895c7c9-6s4x6\" (UID: \"e5640e81-834c-4854-9cc9-589ccbd39d64\") " pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" Nov 02 19:47:43.243782 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.243684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6b95\" (UniqueName: \"kubernetes.io/projected/a36272f8-3fbc-4501-b03c-91a343e37204-kube-api-access-j6b95\") pod \"volume-data-source-validator-58c9f5f46d-55rd9\" (UID: \"a36272f8-3fbc-4501-b03c-91a343e37204\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" Nov 02 19:47:43.245903 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.245881 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6n7fh\"" Nov 02 19:47:43.246186 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.246171 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-27pq5"] Nov 02 19:47:43.246317 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.246301 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.250268 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.250251 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 19:47:43.252451 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.252434 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t6dtl\"" Nov 02 19:47:43.264887 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.264865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.264976 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.264904 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv"] Nov 02 19:47:43.272854 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.272837 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9xjzz\"" Nov 02 19:47:43.272930 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.272860 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 19:47:43.281070 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.281050 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv"] Nov 02 19:47:43.281156 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.281103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.295155 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.295138 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f6668f8f7-2vjcr"] Nov 02 19:47:43.295244 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.295231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.302587 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302560 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.302683 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302601 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.302683 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302624 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 19:47:43.302683 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302628 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.302683 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302650 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.302815 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302703 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.302815 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302712 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 19:47:43.302874 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302865 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.302918 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302900 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 19:47:43.302952 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302920 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.302984 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.302969 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.307337 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.307319 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7756d77bb6-lfwx6"] Nov 02 19:47:43.307449 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.307436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.320412 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.320391 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f"] Nov 02 19:47:43.320508 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.320497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.332876 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.332859 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 19:47:43.334671 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.334657 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pxnkj"] Nov 02 19:47:43.334768 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.334755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" Nov 02 19:47:43.344808 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.344790 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bffd989f9-r4bss"] Nov 02 19:47:43.344896 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.344810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff626920-a094-4782-aae1-ac6bc9d0b359-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.344896 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.344832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-tmp\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.344896 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.344851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.345048 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.344907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:43.345048 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.344916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97562254-d922-4415-9438-ae340b218942-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.345048 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.344969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.345048 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b95\" (UniqueName: \"kubernetes.io/projected/a36272f8-3fbc-4501-b03c-91a343e37204-kube-api-access-j6b95\") pod \"volume-data-source-validator-58c9f5f46d-55rd9\" (UID: \"a36272f8-3fbc-4501-b03c-91a343e37204\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" Nov 02 19:47:43.345048 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-service-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.345269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fb381a6-0d14-47a9-a144-4fd8be651947-serving-cert\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.345269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.345269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjmr\" (UniqueName: \"kubernetes.io/projected/703df830-74cd-46d0-96ab-ab6b2b42c439-kube-api-access-jnjmr\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.345269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn7xb\" (UniqueName: \"kubernetes.io/projected/97562254-d922-4415-9438-ae340b218942-kube-api-access-tn7xb\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.345503 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff626920-a094-4782-aae1-ac6bc9d0b359-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.345503 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.345503 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct6x2\" (UniqueName: \"kubernetes.io/projected/ff626920-a094-4782-aae1-ac6bc9d0b359-kube-api-access-ct6x2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.345503 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-snapshots\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.345503 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a620cde-dc26-4159-8e40-41fdd982d61f-config-volume\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deaf3243-1eeb-4d94-bef9-524b67035ba2-serving-cert\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-trusted-ca\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggbz\" (UniqueName: \"kubernetes.io/projected/f98defbd-aa11-4a76-bac3-4d7aacdb44a1-kube-api-access-7ggbz\") pod \"kas-connection-checker-685895c7c9-t25fz\" (UID: \"f98defbd-aa11-4a76-bac3-4d7aacdb44a1\") " pod="kube-system/kas-connection-checker-685895c7c9-t25fz" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcnh\" (UniqueName: \"kubernetes.io/projected/cb5fc542-006a-4da5-b985-cc7b0f1709a7-kube-api-access-8mcnh\") pod \"kas-connection-checker-685895c7c9-qwntm\" (UID: \"cb5fc542-006a-4da5-b985-cc7b0f1709a7\") " pod="kube-system/kas-connection-checker-685895c7c9-qwntm" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdr6s\" (UniqueName: \"kubernetes.io/projected/deaf3243-1eeb-4d94-bef9-524b67035ba2-kube-api-access-sdr6s\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-config\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.345721 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a620cde-dc26-4159-8e40-41fdd982d61f-tmp-dir\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.346112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnqz\" (UniqueName: \"kubernetes.io/projected/1a620cde-dc26-4159-8e40-41fdd982d61f-kube-api-access-ffnqz\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.346112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9m9j\" (UniqueName: \"kubernetes.io/projected/e5640e81-834c-4854-9cc9-589ccbd39d64-kube-api-access-s9m9j\") pod \"kas-connection-checker-685895c7c9-6s4x6\" (UID: \"e5640e81-834c-4854-9cc9-589ccbd39d64\") " pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" Nov 02 19:47:43.346112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.345830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsfs\" (UniqueName: \"kubernetes.io/projected/0fb381a6-0d14-47a9-a144-4fd8be651947-kube-api-access-wfsfs\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.354599 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.354578 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:47:43.354687 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.354636 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wj4w6\"" Nov 02 19:47:43.354687 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.354681 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pwftm\"" Nov 02 19:47:43.354799 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.354689 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 19:47:43.354799 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.354723 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 19:47:43.354799 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.354749 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 19:47:43.355014 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.355000 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 19:47:43.356275 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.356259 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7cqmh\"" Nov 02 19:47:43.356349 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.356303 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h9lqd\"" Nov 02 19:47:43.356349 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.356303 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 19:47:43.356349 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.356320 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vvz96\"" Nov 02 19:47:43.357904 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.357890 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dkb72"] Nov 02 19:47:43.358014 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.358001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.362138 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.362097 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 19:47:43.362138 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.362108 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 19:47:43.362269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.362140 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mg6c7\"" Nov 02 19:47:43.362269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.362163 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 19:47:43.368091 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368075 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-t25fz"] Nov 02 19:47:43.368170 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368104 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qwntm"] Nov 02 19:47:43.368170 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368118 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6s4x6"] Nov 02 19:47:43.368170 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368142 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9"] Nov 02 19:47:43.368170 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368156 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7"] Nov 02 19:47:43.368170 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368170 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv"] Nov 02 19:47:43.368332 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368184 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kg2ng"] Nov 02 19:47:43.368332 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368198 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-27pq5"] Nov 02 19:47:43.368332 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368220 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f"] Nov 02 19:47:43.368332 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368232 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pxnkj"] Nov 02 19:47:43.368332 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368246 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7756d77bb6-lfwx6"] Nov 02 19:47:43.368332 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.368332 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368259 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc"] Nov 02 19:47:43.368541 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368336 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dkb72"] Nov 02 19:47:43.368541 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368353 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv"] Nov 02 19:47:43.368541 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368369 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2b245"] Nov 02 19:47:43.368541 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368382 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f6668f8f7-2vjcr"] Nov 02 19:47:43.368541 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.368402 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bffd989f9-r4bss"] Nov 02 19:47:43.373933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.373915 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 19:47:43.374473 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.374456 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5cssh\"" Nov 02 19:47:43.374558 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.374488 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 19:47:43.374558 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.374493 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 19:47:43.387898 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.387883 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q7g7q\"" Nov 02 19:47:43.398929 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.398905 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 19:47:43.399009 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.398948 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-869qp\"" Nov 02 19:47:43.405452 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405430 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 19:47:43.405561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405484 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.405561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405528 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.405671 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405631 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 19:47:43.405723 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405672 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.405723 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405706 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 19:47:43.405826 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405699 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.405826 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405750 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 19:47:43.405921 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405828 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.405921 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405860 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.405921 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.405917 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:47:43.406101 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406083 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.406194 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406085 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.406194 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406087 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.406376 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406120 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 19:47:43.406376 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406140 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.406376 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406372 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggbz\" (UniqueName: \"kubernetes.io/projected/f98defbd-aa11-4a76-bac3-4d7aacdb44a1-kube-api-access-7ggbz\") pod \"kas-connection-checker-685895c7c9-t25fz\" (UID: \"f98defbd-aa11-4a76-bac3-4d7aacdb44a1\") " pod="kube-system/kas-connection-checker-685895c7c9-t25fz" Nov 02 19:47:43.406527 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b95\" (UniqueName: \"kubernetes.io/projected/a36272f8-3fbc-4501-b03c-91a343e37204-kube-api-access-j6b95\") pod \"volume-data-source-validator-58c9f5f46d-55rd9\" (UID: \"a36272f8-3fbc-4501-b03c-91a343e37204\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" Nov 02 19:47:43.406670 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.406645 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcnh\" (UniqueName: \"kubernetes.io/projected/cb5fc542-006a-4da5-b985-cc7b0f1709a7-kube-api-access-8mcnh\") pod \"kas-connection-checker-685895c7c9-qwntm\" (UID: \"cb5fc542-006a-4da5-b985-cc7b0f1709a7\") " pod="kube-system/kas-connection-checker-685895c7c9-qwntm" Nov 02 19:47:43.407323 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.407305 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 19:47:43.407384 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.407320 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9m9j\" (UniqueName: \"kubernetes.io/projected/e5640e81-834c-4854-9cc9-589ccbd39d64-kube-api-access-s9m9j\") pod \"kas-connection-checker-685895c7c9-6s4x6\" (UID: \"e5640e81-834c-4854-9cc9-589ccbd39d64\") " pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" Nov 02 19:47:43.422178 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.422158 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:47:43.422269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.422175 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:47:43.422269 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.422184 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:47:43.422364 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.422318 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t7wfl\"" Nov 02 19:47:43.446580 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-trusted-ca\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.446672 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:43.446672 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47cn\" (UniqueName: \"kubernetes.io/projected/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-kube-api-access-r47cn\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.446672 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.446672 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-config\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.446795 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-stats-auth\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.446870 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsfs\" (UniqueName: \"kubernetes.io/projected/0fb381a6-0d14-47a9-a144-4fd8be651947-kube-api-access-wfsfs\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.446915 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.446950 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-service-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.446986 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fb381a6-0d14-47a9-a144-4fd8be651947-serving-cert\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.447038 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.446982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.447038 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7xb\" (UniqueName: \"kubernetes.io/projected/97562254-d922-4415-9438-ae340b218942-kube-api-access-tn7xb\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.447131 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.447131 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.447233 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff626920-a094-4782-aae1-ac6bc9d0b359-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.447233 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.447233 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.447233 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct6x2\" (UniqueName: \"kubernetes.io/projected/ff626920-a094-4782-aae1-ac6bc9d0b359-kube-api-access-ct6x2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.447455 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.447238 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:47:43.447455 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.447313 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls podName:703df830-74cd-46d0-96ab-ab6b2b42c439 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:43.947277108 +0000 UTC m=+36.763272922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wvrhv" (UID: "703df830-74cd-46d0-96ab-ab6b2b42c439") : secret "samples-operator-tls" not found Nov 02 19:47:43.447455 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b79f192-7b15-47d5-91b4-1772b3541ecf-cert\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.447455 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jckjq\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.447455 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447393 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-config\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.447455 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdr6s\" (UniqueName: \"kubernetes.io/projected/deaf3243-1eeb-4d94-bef9-524b67035ba2-kube-api-access-sdr6s\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447539 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-service-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a620cde-dc26-4159-8e40-41fdd982d61f-tmp-dir\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnqz\" (UniqueName: \"kubernetes.io/projected/1a620cde-dc26-4159-8e40-41fdd982d61f-kube-api-access-ffnqz\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-trusted-ca\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deaf3243-1eeb-4d94-bef9-524b67035ba2-serving-cert\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-trusted-ca\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.447767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-image-registry-private-configuration\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-installation-pull-secrets\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-config\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff626920-a094-4782-aae1-ac6bc9d0b359-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a620cde-dc26-4159-8e40-41fdd982d61f-tmp-dir\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.447888 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.447935 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls podName:1a620cde-dc26-4159-8e40-41fdd982d61f nodeName:}" failed. No retries permitted until 2026-11-02 19:47:43.947918965 +0000 UTC m=+36.763914766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls") pod "dns-default-27pq5" (UID: "1a620cde-dc26-4159-8e40-41fdd982d61f") : secret "dns-default-metrics-tls" not found Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.447955 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff626920-a094-4782-aae1-ac6bc9d0b359-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.447992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-tmp\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.448048 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls podName:97562254-d922-4415-9438-ae340b218942 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:43.948035678 +0000 UTC m=+36.764031483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kz9f7" (UID: "97562254-d922-4415-9438-ae340b218942") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97562254-d922-4415-9438-ae340b218942-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.448206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2419a0c9-85c6-417d-9c5a-358e533246a9-ca-trust-extracted\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxmg\" (UniqueName: \"kubernetes.io/projected/3b79f192-7b15-47d5-91b4-1772b3541ecf-kube-api-access-6cxmg\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-certificates\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-serving-cert\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-default-certificate\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448436 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-tmp\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtw5\" (UniqueName: \"kubernetes.io/projected/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-kube-api-access-jhtw5\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8b7\" (UniqueName: \"kubernetes.io/projected/8faa206c-23f9-4f03-8b85-d66e47f08c30-kube-api-access-lm8b7\") pod \"network-check-source-6f78d59f99-95m8f\" (UID: \"8faa206c-23f9-4f03-8b85-d66e47f08c30\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-bound-sa-token\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-snapshots\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a620cde-dc26-4159-8e40-41fdd982d61f-config-volume\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9271574f-8b74-4de7-8bcf-f90951a5fd4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.448934 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.449637 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.448992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97562254-d922-4415-9438-ae340b218942-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.449637 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.449257 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-snapshots\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.449637 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.449270 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a620cde-dc26-4159-8e40-41fdd982d61f-config-volume\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.449637 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.449342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66qrm\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-kube-api-access-66qrm\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.449637 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.449365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjmr\" (UniqueName: \"kubernetes.io/projected/703df830-74cd-46d0-96ab-ab6b2b42c439-kube-api-access-jnjmr\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.450006 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.449986 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fb381a6-0d14-47a9-a144-4fd8be651947-serving-cert\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.450230 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.450211 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deaf3243-1eeb-4d94-bef9-524b67035ba2-serving-cert\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.450410 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.450395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff626920-a094-4782-aae1-ac6bc9d0b359-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.455642 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.455627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-t25fz" Nov 02 19:47:43.468360 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.468340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qwntm" Nov 02 19:47:43.490656 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.490593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" Nov 02 19:47:43.505868 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.505721 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsfs\" (UniqueName: \"kubernetes.io/projected/0fb381a6-0d14-47a9-a144-4fd8be651947-kube-api-access-wfsfs\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.508612 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.508584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" Nov 02 19:47:43.509162 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.509141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnqz\" (UniqueName: \"kubernetes.io/projected/1a620cde-dc26-4159-8e40-41fdd982d61f-kube-api-access-ffnqz\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.509401 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.509382 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct6x2\" (UniqueName: \"kubernetes.io/projected/ff626920-a094-4782-aae1-ac6bc9d0b359-kube-api-access-ct6x2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.509470 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.509412 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdr6s\" (UniqueName: \"kubernetes.io/projected/deaf3243-1eeb-4d94-bef9-524b67035ba2-kube-api-access-sdr6s\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.509470 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.509453 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjmr\" (UniqueName: \"kubernetes.io/projected/703df830-74cd-46d0-96ab-ab6b2b42c439-kube-api-access-jnjmr\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.509642 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.509626 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7xb\" (UniqueName: \"kubernetes.io/projected/97562254-d922-4415-9438-ae340b218942-kube-api-access-tn7xb\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.542869 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.542823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:47:43.550049 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2419a0c9-85c6-417d-9c5a-358e533246a9-ca-trust-extracted\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.550144 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxmg\" (UniqueName: \"kubernetes.io/projected/3b79f192-7b15-47d5-91b4-1772b3541ecf-kube-api-access-6cxmg\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.550144 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-certificates\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.550144 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-serving-cert\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.550144 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-default-certificate\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.550343 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.550343 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.550254 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:47:43.550343 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.550341 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.050320492 +0000 UTC m=+36.866316306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : secret "router-metrics-certs-default" not found Nov 02 19:47:43.550479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtw5\" (UniqueName: \"kubernetes.io/projected/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-kube-api-access-jhtw5\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.550479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8b7\" (UniqueName: \"kubernetes.io/projected/8faa206c-23f9-4f03-8b85-d66e47f08c30-kube-api-access-lm8b7\") pod \"network-check-source-6f78d59f99-95m8f\" (UID: \"8faa206c-23f9-4f03-8b85-d66e47f08c30\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" Nov 02 19:47:43.550479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-bound-sa-token\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.550479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9271574f-8b74-4de7-8bcf-f90951a5fd4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:43.550479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qrm\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-kube-api-access-66qrm\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47cn\" (UniqueName: \"kubernetes.io/projected/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-kube-api-access-r47cn\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-stats-auth\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.550748 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.551050 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.551050 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.551050 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-certificates\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.551050 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.550847 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:47:43.551050 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.550911 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert podName:9271574f-8b74-4de7-8bcf-f90951a5fd4a nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.050895877 +0000 UTC m=+36.866891694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxnkj" (UID: "9271574f-8b74-4de7-8bcf-f90951a5fd4a") : secret "networking-console-plugin-cert" not found Nov 02 19:47:43.551410 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.551386 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.051370189 +0000 UTC m=+36.867366018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : configmap references non-existent config key: service-ca.crt Nov 02 19:47:43.551585 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.551561 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2419a0c9-85c6-417d-9c5a-358e533246a9-ca-trust-extracted\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.551722 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.551706 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:43.551795 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.551725 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bffd989f9-r4bss: secret "image-registry-tls" not found Nov 02 19:47:43.551795 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.551757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9271574f-8b74-4de7-8bcf-f90951a5fd4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:43.551795 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.551794 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls podName:fe5633bb-4256-4685-943e-0e0ecbce30cd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.051778155 +0000 UTC m=+36.867773959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls") pod "image-registry-5bffd989f9-r4bss" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd") : secret "image-registry-tls" not found Nov 02 19:47:43.552252 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.552231 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.552796 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.550824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b79f192-7b15-47d5-91b4-1772b3541ecf-cert\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.552946 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.552928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckjq\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.553069 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.553056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.553134 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.553126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-trusted-ca\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.553198 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.553189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.553253 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.553245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-image-registry-private-configuration\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.553353 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.553338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-installation-pull-secrets\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.553457 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.553441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-config\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.554258 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.554239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-config\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.554942 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.554916 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.555044 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.555027 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:43.555044 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.555032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.555150 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.555048 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7756d77bb6-lfwx6: secret "image-registry-tls" not found Nov 02 19:47:43.555150 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.555088 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls podName:2419a0c9-85c6-417d-9c5a-358e533246a9 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.055074389 +0000 UTC m=+36.871070208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls") pod "image-registry-7756d77bb6-lfwx6" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9") : secret "image-registry-tls" not found Nov 02 19:47:43.555631 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.555451 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-serving-cert\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.557565 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.555961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-stats-auth\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.557565 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.556171 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.557565 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.556533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-trusted-ca\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.557565 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.556694 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-default-certificate\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.557761 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.557611 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b79f192-7b15-47d5-91b4-1772b3541ecf-cert\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.558222 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.558200 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-installation-pull-secrets\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.558457 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.558438 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.559574 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.559555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-image-registry-private-configuration\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.565709 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.565689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:47:43.587362 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.587328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:43.623429 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.623395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckjq\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.629981 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.629435 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:43.629981 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.629904 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8b7\" (UniqueName: \"kubernetes.io/projected/8faa206c-23f9-4f03-8b85-d66e47f08c30-kube-api-access-lm8b7\") pod \"network-check-source-6f78d59f99-95m8f\" (UID: \"8faa206c-23f9-4f03-8b85-d66e47f08c30\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" Nov 02 19:47:43.629981 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.629960 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-bound-sa-token\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.631614 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.630626 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qrm\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-kube-api-access-66qrm\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:43.631614 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.630959 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtw5\" (UniqueName: \"kubernetes.io/projected/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-kube-api-access-jhtw5\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.631614 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.631530 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxmg\" (UniqueName: \"kubernetes.io/projected/3b79f192-7b15-47d5-91b4-1772b3541ecf-kube-api-access-6cxmg\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.631614 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.631566 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47cn\" (UniqueName: \"kubernetes.io/projected/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-kube-api-access-r47cn\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:43.690430 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.690334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" Nov 02 19:47:43.756081 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.755888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:47:43.927489 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.927457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:47:43.932159 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:43.932126 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5fc542_006a_4da5_b985_cc7b0f1709a7.slice/crio-d7889132659b1013de4f59658e7579684be8afe85340621477345acf2ec1d376 WatchSource:0}: Error finding container d7889132659b1013de4f59658e7579684be8afe85340621477345acf2ec1d376: Status 404 returned error can't find the container with id d7889132659b1013de4f59658e7579684be8afe85340621477345acf2ec1d376 Nov 02 19:47:43.933253 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.933221 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qwntm"] Nov 02 19:47:43.933985 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:43.933965 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff626920_a094_4782_aae1_ac6bc9d0b359.slice/crio-436ecc028f0783e87e73964081c3450fa555f6c621e23cfa51d56d65404575ef WatchSource:0}: Error finding container 436ecc028f0783e87e73964081c3450fa555f6c621e23cfa51d56d65404575ef: Status 404 returned error can't find the container with id 436ecc028f0783e87e73964081c3450fa555f6c621e23cfa51d56d65404575ef Nov 02 19:47:43.935686 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:43.935660 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36272f8_3fbc_4501_b03c_91a343e37204.slice/crio-d4c2b765b7a1efaa8ef7994476dec85ed89b4270b3433294f2412e65008050ab WatchSource:0}: Error finding container d4c2b765b7a1efaa8ef7994476dec85ed89b4270b3433294f2412e65008050ab: Status 404 returned error can't find the container with id d4c2b765b7a1efaa8ef7994476dec85ed89b4270b3433294f2412e65008050ab Nov 02 19:47:43.936746 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.936718 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9"] Nov 02 19:47:43.937767 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.937747 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc"] Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.957266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.957363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.957398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.957407 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.957473 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls podName:703df830-74cd-46d0-96ab-ab6b2b42c439 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.957453718 +0000 UTC m=+37.773449537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wvrhv" (UID: "703df830-74cd-46d0-96ab-ab6b2b42c439") : secret "samples-operator-tls" not found Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.957525 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.957576 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls podName:97562254-d922-4415-9438-ae340b218942 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.957559067 +0000 UTC m=+37.773554881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kz9f7" (UID: "97562254-d922-4415-9438-ae340b218942") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:43.957629 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.957623 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:47:43.958077 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:43.957660 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls podName:1a620cde-dc26-4159-8e40-41fdd982d61f nodeName:}" failed. No retries permitted until 2026-11-02 19:47:44.957647505 +0000 UTC m=+37.773643317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls") pod "dns-default-27pq5" (UID: "1a620cde-dc26-4159-8e40-41fdd982d61f") : secret "dns-default-metrics-tls" not found Nov 02 19:47:43.978192 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:43.978155 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98defbd_aa11_4a76_bac3_4d7aacdb44a1.slice/crio-4455d1c23aed6d355149b9f165eb7fd81392b806f926ca925d76203776d7886b WatchSource:0}: Error finding container 4455d1c23aed6d355149b9f165eb7fd81392b806f926ca925d76203776d7886b: Status 404 returned error can't find the container with id 4455d1c23aed6d355149b9f165eb7fd81392b806f926ca925d76203776d7886b Nov 02 19:47:43.978799 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.978777 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-t25fz"] Nov 02 19:47:43.981494 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:43.981392 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5640e81_834c_4854_9cc9_589ccbd39d64.slice/crio-ea3430cdb385a0478c77b8bcc290714b265165e826ce14a7e16e729bd6641f54 WatchSource:0}: Error finding container ea3430cdb385a0478c77b8bcc290714b265165e826ce14a7e16e729bd6641f54: Status 404 returned error can't find the container with id ea3430cdb385a0478c77b8bcc290714b265165e826ce14a7e16e729bd6641f54 Nov 02 19:47:43.982981 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:43.982652 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6s4x6"] Nov 02 19:47:44.028612 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.028536 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-2b245"] Nov 02 19:47:44.034259 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.034160 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kg2ng"] Nov 02 19:47:44.053085 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:44.053054 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb381a6_0d14_47a9_a144_4fd8be651947.slice/crio-ad7ee0d0e6f7758a48042a1fafd594eb8d12952971209634878c3dc1817e7e45 WatchSource:0}: Error finding container ad7ee0d0e6f7758a48042a1fafd594eb8d12952971209634878c3dc1817e7e45: Status 404 returned error can't find the container with id ad7ee0d0e6f7758a48042a1fafd594eb8d12952971209634878c3dc1817e7e45 Nov 02 19:47:44.059056 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.059030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:44.059158 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.059098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:44.059158 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.059135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:44.059277 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.059167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:44.059277 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.059256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:45.05941369 +0000 UTC m=+37.875409513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : configmap references non-existent config key: service-ca.crt Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059520 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059523 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059544 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bffd989f9-r4bss: secret "image-registry-tls" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059585 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059596 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls podName:fe5633bb-4256-4685-943e-0e0ecbce30cd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:45.059578677 +0000 UTC m=+37.875574497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls") pod "image-registry-5bffd989f9-r4bss" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd") : secret "image-registry-tls" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059621 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:45.059608792 +0000 UTC m=+37.875604609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : secret "router-metrics-certs-default" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059532 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7756d77bb6-lfwx6: secret "image-registry-tls" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059654 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls podName:2419a0c9-85c6-417d-9c5a-358e533246a9 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:45.059645242 +0000 UTC m=+37.875641056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls") pod "image-registry-7756d77bb6-lfwx6" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9") : secret "image-registry-tls" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059662 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:47:44.059721 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.059696 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert podName:9271574f-8b74-4de7-8bcf-f90951a5fd4a nodeName:}" failed. No retries permitted until 2026-11-02 19:47:45.059685834 +0000 UTC m=+37.875681659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxnkj" (UID: "9271574f-8b74-4de7-8bcf-f90951a5fd4a") : secret "networking-console-plugin-cert" not found Nov 02 19:47:44.083773 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:44.083730 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faa206c_23f9_4f03_8b85_d66e47f08c30.slice/crio-22ef7c50f8e0f13b678ac9707f06ae17ddc76632dc960700e715f04f44e58aac WatchSource:0}: Error finding container 22ef7c50f8e0f13b678ac9707f06ae17ddc76632dc960700e715f04f44e58aac: Status 404 returned error can't find the container with id 22ef7c50f8e0f13b678ac9707f06ae17ddc76632dc960700e715f04f44e58aac Nov 02 19:47:44.085937 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.085485 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f"] Nov 02 19:47:44.149575 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:44.149552 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b79f192_7b15_47d5_91b4_1772b3541ecf.slice/crio-483d11b57a7c5b92a6162154b952c246d5da36af585db874389d0091d1d96c61 WatchSource:0}: Error finding container 483d11b57a7c5b92a6162154b952c246d5da36af585db874389d0091d1d96c61: Status 404 returned error can't find the container with id 483d11b57a7c5b92a6162154b952c246d5da36af585db874389d0091d1d96c61 Nov 02 19:47:44.151777 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.151758 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dkb72"] Nov 02 19:47:44.344962 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:44.344934 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b42a16_a3fd_4eeb_8fbe_cb1b5a40fd6f.slice/crio-886eee9e6338a661d858d979d3ef04f641715fa875aa1e1c99d7221445275b1a WatchSource:0}: Error finding container 886eee9e6338a661d858d979d3ef04f641715fa875aa1e1c99d7221445275b1a: Status 404 returned error can't find the container with id 886eee9e6338a661d858d979d3ef04f641715fa875aa1e1c99d7221445275b1a Nov 02 19:47:44.346727 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.346677 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv"] Nov 02 19:47:44.643889 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.643814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:47:44.643889 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.643849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:47:44.644082 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.643905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:47:44.697458 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.697390 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:47:44.697905 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.697632 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\"" Nov 02 19:47:44.697905 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.697657 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\"" Nov 02 19:47:44.697905 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.697700 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\"" Nov 02 19:47:44.697905 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.697750 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:47:44.858920 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.858879 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" event={"ID":"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f","Type":"ContainerStarted","Data":"886eee9e6338a661d858d979d3ef04f641715fa875aa1e1c99d7221445275b1a"} Nov 02 19:47:44.859937 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.859915 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kg2ng" event={"ID":"0fb381a6-0d14-47a9-a144-4fd8be651947","Type":"ContainerStarted","Data":"ad7ee0d0e6f7758a48042a1fafd594eb8d12952971209634878c3dc1817e7e45"} Nov 02 19:47:44.861034 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.861013 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dkb72" event={"ID":"3b79f192-7b15-47d5-91b4-1772b3541ecf","Type":"ContainerStarted","Data":"483d11b57a7c5b92a6162154b952c246d5da36af585db874389d0091d1d96c61"} Nov 02 19:47:44.862192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.862170 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" event={"ID":"8faa206c-23f9-4f03-8b85-d66e47f08c30","Type":"ContainerStarted","Data":"22ef7c50f8e0f13b678ac9707f06ae17ddc76632dc960700e715f04f44e58aac"} Nov 02 19:47:44.863331 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.863308 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" event={"ID":"deaf3243-1eeb-4d94-bef9-524b67035ba2","Type":"ContainerStarted","Data":"cf0e2eb10c8cec711f8c1fba04e0fbc64030fe157ddf5f03312f81cee70aac4a"} Nov 02 19:47:44.864941 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.864902 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" event={"ID":"e5640e81-834c-4854-9cc9-589ccbd39d64","Type":"ContainerStarted","Data":"e5b046ee7db56da6c187901935ef64afbdaa5f13eb054bd80ad1b85f33c2d6ac"} Nov 02 19:47:44.865040 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.864989 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" event={"ID":"e5640e81-834c-4854-9cc9-589ccbd39d64","Type":"ContainerStarted","Data":"ea3430cdb385a0478c77b8bcc290714b265165e826ce14a7e16e729bd6641f54"} Nov 02 19:47:44.866033 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.866002 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" event={"ID":"ff626920-a094-4782-aae1-ac6bc9d0b359","Type":"ContainerStarted","Data":"436ecc028f0783e87e73964081c3450fa555f6c621e23cfa51d56d65404575ef"} Nov 02 19:47:44.867470 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.867450 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qwntm" event={"ID":"cb5fc542-006a-4da5-b985-cc7b0f1709a7","Type":"ContainerStarted","Data":"998614ceb7d41e7a12feffd9de1b06a7a3f32724353938cdac04d7b84928b5c1"} Nov 02 19:47:44.867621 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.867487 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qwntm" event={"ID":"cb5fc542-006a-4da5-b985-cc7b0f1709a7","Type":"ContainerStarted","Data":"d7889132659b1013de4f59658e7579684be8afe85340621477345acf2ec1d376"} Nov 02 19:47:44.868805 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.868784 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" event={"ID":"a36272f8-3fbc-4501-b03c-91a343e37204","Type":"ContainerStarted","Data":"d4c2b765b7a1efaa8ef7994476dec85ed89b4270b3433294f2412e65008050ab"} Nov 02 19:47:44.870370 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.870338 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-t25fz" event={"ID":"f98defbd-aa11-4a76-bac3-4d7aacdb44a1","Type":"ContainerStarted","Data":"85fdf76719259501d482d2d6771057e67f85913e2ab5abfb78609ae0f7d4d5fb"} Nov 02 19:47:44.870437 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.870375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-t25fz" event={"ID":"f98defbd-aa11-4a76-bac3-4d7aacdb44a1","Type":"ContainerStarted","Data":"4455d1c23aed6d355149b9f165eb7fd81392b806f926ca925d76203776d7886b"} Nov 02 19:47:44.967386 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.967305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:44.967561 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.967404 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:47:44.967561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.967411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:44.967561 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.967474 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls podName:703df830-74cd-46d0-96ab-ab6b2b42c439 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:46.967455112 +0000 UTC m=+39.783450926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wvrhv" (UID: "703df830-74cd-46d0-96ab-ab6b2b42c439") : secret "samples-operator-tls" not found Nov 02 19:47:44.967561 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.967495 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:47:44.967561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:44.967529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:44.967561 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.967552 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls podName:1a620cde-dc26-4159-8e40-41fdd982d61f nodeName:}" failed. No retries permitted until 2026-11-02 19:47:46.967538845 +0000 UTC m=+39.783534660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls") pod "dns-default-27pq5" (UID: "1a620cde-dc26-4159-8e40-41fdd982d61f") : secret "dns-default-metrics-tls" not found Nov 02 19:47:44.967837 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.967591 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:44.967837 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:44.967641 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls podName:97562254-d922-4415-9438-ae340b218942 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:46.967629842 +0000 UTC m=+39.783625660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kz9f7" (UID: "97562254-d922-4415-9438-ae340b218942") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:45.068851 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.068810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.068868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.068902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.068977 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.068999 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bffd989f9-r4bss: secret "image-registry-tls" not found Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.069009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069035 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069049 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7756d77bb6-lfwx6: secret "image-registry-tls" not found Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069058 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls podName:fe5633bb-4256-4685-943e-0e0ecbce30cd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:47.069038831 +0000 UTC m=+39.885034645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls") pod "image-registry-5bffd989f9-r4bss" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd") : secret "image-registry-tls" not found Nov 02 19:47:45.069104 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069093 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls podName:2419a0c9-85c6-417d-9c5a-358e533246a9 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:47.069078585 +0000 UTC m=+39.885074399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls") pod "image-registry-7756d77bb6-lfwx6" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9") : secret "image-registry-tls" not found Nov 02 19:47:45.069564 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069113 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:47.069100594 +0000 UTC m=+39.885096413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : configmap references non-existent config key: service-ca.crt Nov 02 19:47:45.069564 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069146 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:47:45.069564 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.069223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:45.069564 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069260 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert podName:9271574f-8b74-4de7-8bcf-f90951a5fd4a nodeName:}" failed. No retries permitted until 2026-11-02 19:47:47.069244212 +0000 UTC m=+39.885240024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxnkj" (UID: "9271574f-8b74-4de7-8bcf-f90951a5fd4a") : secret "networking-console-plugin-cert" not found Nov 02 19:47:45.069564 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069316 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:47:45.069564 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:45.069354 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:47.069341402 +0000 UTC m=+39.885337207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : secret "router-metrics-certs-default" not found Nov 02 19:47:45.227824 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.227721 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qwntm" podStartSLOduration=452.227701932 podStartE2EDuration="7m32.227701932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:45.22637501 +0000 UTC m=+38.042370832" watchObservedRunningTime="2026-11-02 19:47:45.227701932 +0000 UTC m=+38.043697758" Nov 02 19:47:45.348871 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.348562 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-t25fz" podStartSLOduration=452.348544179 podStartE2EDuration="7m32.348544179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:45.346571157 +0000 UTC m=+38.162566979" watchObservedRunningTime="2026-11-02 19:47:45.348544179 +0000 UTC m=+38.164540003" Nov 02 19:47:45.462860 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:45.462616 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" podStartSLOduration=452.462594872 podStartE2EDuration="7m32.462594872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:45.460815168 +0000 UTC m=+38.276810994" watchObservedRunningTime="2026-11-02 19:47:45.462594872 +0000 UTC m=+38.278590695" Nov 02 19:47:46.887640 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:46.887558 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" event={"ID":"a36272f8-3fbc-4501-b03c-91a343e37204","Type":"ContainerStarted","Data":"dbf5dffeb9a6423047841920c95b9bacbe7576743e50cc21cdd7355348a6d93d"} Nov 02 19:47:46.991550 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:46.991508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:46.991713 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:46.991618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:46.991713 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:46.991660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:46.991950 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:46.991916 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:47:46.991950 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:46.991939 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:47:46.992321 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:46.992305 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:46.993449 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:46.992501 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls podName:703df830-74cd-46d0-96ab-ab6b2b42c439 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:50.992479528 +0000 UTC m=+43.808475328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wvrhv" (UID: "703df830-74cd-46d0-96ab-ab6b2b42c439") : secret "samples-operator-tls" not found Nov 02 19:47:46.993449 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:46.992529 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls podName:1a620cde-dc26-4159-8e40-41fdd982d61f nodeName:}" failed. No retries permitted until 2026-11-02 19:47:50.992519615 +0000 UTC m=+43.808515417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls") pod "dns-default-27pq5" (UID: "1a620cde-dc26-4159-8e40-41fdd982d61f") : secret "dns-default-metrics-tls" not found Nov 02 19:47:46.993449 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:46.992558 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls podName:97562254-d922-4415-9438-ae340b218942 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:50.992548931 +0000 UTC m=+43.808544734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kz9f7" (UID: "97562254-d922-4415-9438-ae340b218942") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:47.093952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:47.094053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:47.094091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:47.094124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:47.094196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094417 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:51.094393032 +0000 UTC m=+43.910388850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : configmap references non-existent config key: service-ca.crt Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094499 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094532 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:51.094521269 +0000 UTC m=+43.910517093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : secret "router-metrics-certs-default" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094601 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094613 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bffd989f9-r4bss: secret "image-registry-tls" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094642 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls podName:fe5633bb-4256-4685-943e-0e0ecbce30cd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:51.094632653 +0000 UTC m=+43.910628468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls") pod "image-registry-5bffd989f9-r4bss" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd") : secret "image-registry-tls" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094698 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094729 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert podName:9271574f-8b74-4de7-8bcf-f90951a5fd4a nodeName:}" failed. No retries permitted until 2026-11-02 19:47:51.094718555 +0000 UTC m=+43.910714369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxnkj" (UID: "9271574f-8b74-4de7-8bcf-f90951a5fd4a") : secret "networking-console-plugin-cert" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094784 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094794 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7756d77bb6-lfwx6: secret "image-registry-tls" not found Nov 02 19:47:47.094852 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:47.094821 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls podName:2419a0c9-85c6-417d-9c5a-358e533246a9 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:51.094811415 +0000 UTC m=+43.910807229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls") pod "image-registry-7756d77bb6-lfwx6" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9") : secret "image-registry-tls" not found Nov 02 19:47:49.901413 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:49.900977 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" event={"ID":"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f","Type":"ContainerStarted","Data":"df497ecc58c812385314dfab5c4d2bc33e784ea26d82899100156df015df60b2"} Nov 02 19:47:49.903059 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:49.903000 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kg2ng" event={"ID":"0fb381a6-0d14-47a9-a144-4fd8be651947","Type":"ContainerStarted","Data":"f440c72623dba03fdb7a5ef29a0f045f77797c0557980910c67e6acf3391aab8"} Nov 02 19:47:49.904842 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:49.904813 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/0.log" Nov 02 19:47:49.904938 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:49.904849 2579 generic.go:356] "Generic (PLEG): container finished" podID="deaf3243-1eeb-4d94-bef9-524b67035ba2" containerID="6badff54e2ea8d5cc947d05e321f1a46be712a39e3336b5423eeb9b70c758ef7" exitCode=255 Nov 02 19:47:49.904938 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:49.904912 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" event={"ID":"deaf3243-1eeb-4d94-bef9-524b67035ba2","Type":"ContainerDied","Data":"6badff54e2ea8d5cc947d05e321f1a46be712a39e3336b5423eeb9b70c758ef7"} Nov 02 19:47:49.905202 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:49.905187 2579 scope.go:122] "RemoveContainer" containerID="6badff54e2ea8d5cc947d05e321f1a46be712a39e3336b5423eeb9b70c758ef7" Nov 02 19:47:49.907054 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:49.907034 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" event={"ID":"ff626920-a094-4782-aae1-ac6bc9d0b359","Type":"ContainerStarted","Data":"6e3d7e340191b99edfd9108e3b57f2c6705e1de039b067c15db8d212cf812b3c"} Nov 02 19:47:50.019787 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.019719 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" podStartSLOduration=387.681272495 podStartE2EDuration="6m30.019703871s" totalImagesPullingTime="2.338431376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:47.011132625 +0000 UTC m=+39.827128449" watchObservedRunningTime="2026-11-02 19:47:50.019703871 +0000 UTC m=+42.835699692" Nov 02 19:47:50.020159 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.020125 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" podStartSLOduration=389.809560455 podStartE2EDuration="6m35.020113368s" totalImagesPullingTime="5.210552913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:50.018989945 +0000 UTC m=+42.834985761" watchObservedRunningTime="2026-11-02 19:47:50.020113368 +0000 UTC m=+42.836109239" Nov 02 19:47:50.136225 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.136182 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" podStartSLOduration=393.533797676 podStartE2EDuration="6m39.136168653s" totalImagesPullingTime="5.602370977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:50.134580419 +0000 UTC m=+42.950576242" watchObservedRunningTime="2026-11-02 19:47:50.136168653 +0000 UTC m=+42.952164471" Nov 02 19:47:50.368736 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.368682 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-kg2ng" podStartSLOduration=383.890267603 podStartE2EDuration="6m29.368667618s" totalImagesPullingTime="5.478400015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:50.368633805 +0000 UTC m=+43.184629628" watchObservedRunningTime="2026-11-02 19:47:50.368667618 +0000 UTC m=+43.184663430" Nov 02 19:47:50.735852 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.735769 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x6hpx"] Nov 02 19:47:50.766712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.766693 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.793901 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.793870 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x6hpx"] Nov 02 19:47:50.845359 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.845329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnnd\" (UniqueName: \"kubernetes.io/projected/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-api-access-rnnnd\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.845533 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.845442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bd90b6c-66f9-4ea8-8f73-2981acce43db-crio-socket\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.845533 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.845483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bd90b6c-66f9-4ea8-8f73-2981acce43db-data-volume\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.845611 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.845566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.845611 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.845594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.896618 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.896573 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:47:50.897042 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.896805 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:47:50.897042 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.896945 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dsl5q\"" Nov 02 19:47:50.914454 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.913756 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dkb72" event={"ID":"3b79f192-7b15-47d5-91b4-1772b3541ecf","Type":"ContainerStarted","Data":"125c011e71f7e4af6bcb9c129ae4f6ae6f6c4453c4488ee53956c2309520fec5"} Nov 02 19:47:50.916030 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.915999 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" event={"ID":"8faa206c-23f9-4f03-8b85-d66e47f08c30","Type":"ContainerStarted","Data":"b43fb50d41c4ddc53a4b17eba317ad067251807189ac45c6684398beceb5200d"} Nov 02 19:47:50.917790 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.917770 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/1.log" Nov 02 19:47:50.918217 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.918202 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/0.log" Nov 02 19:47:50.918320 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.918236 2579 generic.go:356] "Generic (PLEG): container finished" podID="deaf3243-1eeb-4d94-bef9-524b67035ba2" containerID="28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5" exitCode=255 Nov 02 19:47:50.918373 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.918333 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" event={"ID":"deaf3243-1eeb-4d94-bef9-524b67035ba2","Type":"ContainerDied","Data":"28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5"} Nov 02 19:47:50.918405 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.918377 2579 scope.go:122] "RemoveContainer" containerID="6badff54e2ea8d5cc947d05e321f1a46be712a39e3336b5423eeb9b70c758ef7" Nov 02 19:47:50.918649 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.918631 2579 scope.go:122] "RemoveContainer" containerID="28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5" Nov 02 19:47:50.918854 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:50.918833 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2b245_openshift-console-operator(deaf3243-1eeb-4d94-bef9-524b67035ba2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" podUID="deaf3243-1eeb-4d94-bef9-524b67035ba2" Nov 02 19:47:50.946198 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.946034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnnd\" (UniqueName: \"kubernetes.io/projected/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-api-access-rnnnd\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.946320 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.946257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bd90b6c-66f9-4ea8-8f73-2981acce43db-crio-socket\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.946379 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.946345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bd90b6c-66f9-4ea8-8f73-2981acce43db-data-volume\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.946478 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.946448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.946540 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.946492 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bd90b6c-66f9-4ea8-8f73-2981acce43db-crio-socket\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.946607 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.946587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.946842 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.946816 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bd90b6c-66f9-4ea8-8f73-2981acce43db-data-volume\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:50.947110 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:50.947089 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:47:50.947187 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:50.947155 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:51.447136713 +0000 UTC m=+44.263132537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : secret "insights-runtime-extractor-tls" not found Nov 02 19:47:50.947258 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:50.947206 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:47:50.947258 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:50.947245 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:51.447232866 +0000 UTC m=+44.263228680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : configmap "kube-rbac-proxy" not found Nov 02 19:47:50.998161 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:50.998095 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gcdjm-447t7"] Nov 02 19:47:51.006712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.006688 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnnd\" (UniqueName: \"kubernetes.io/projected/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-api-access-rnnnd\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:51.008063 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.008049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.033749 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.033706 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dkb72" podStartSLOduration=2.049726382 podStartE2EDuration="8.03369523s" totalImagesPullingTime="5.983968848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:51.032803625 +0000 UTC m=+43.848799446" watchObservedRunningTime="2026-11-02 19:47:51.03369523 +0000 UTC m=+43.849691052" Nov 02 19:47:51.047432 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.047402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:51.047561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.047486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bh5c\" (UniqueName: \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.047561 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.047540 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 19:47:51.047561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.047547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:51.047705 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.047622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.047705 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.047649 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls podName:703df830-74cd-46d0-96ab-ab6b2b42c439 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.047632308 +0000 UTC m=+51.863628124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wvrhv" (UID: "703df830-74cd-46d0-96ab-ab6b2b42c439") : secret "samples-operator-tls" not found Nov 02 19:47:51.047705 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.047668 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 19:47:51.047705 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.047680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:51.047867 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.047733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.047867 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.047760 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:51.047867 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.047768 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls podName:1a620cde-dc26-4159-8e40-41fdd982d61f nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.047744002 +0000 UTC m=+51.863739817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls") pod "dns-default-27pq5" (UID: "1a620cde-dc26-4159-8e40-41fdd982d61f") : secret "dns-default-metrics-tls" not found Nov 02 19:47:51.047867 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.047804 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls podName:97562254-d922-4415-9438-ae340b218942 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.047792641 +0000 UTC m=+51.863788447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kz9f7" (UID: "97562254-d922-4415-9438-ae340b218942") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:51.048015 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.047879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.056007 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.055986 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gcdjm-447t7"] Nov 02 19:47:51.148561 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.148526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.148741 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.148580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:51.148741 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.148611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.148741 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.148637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:51.148741 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.148671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148743 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148764 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bffd989f9-r4bss: secret "image-registry-tls" not found Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148765 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.148742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148824 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert podName:9271574f-8b74-4de7-8bcf-f90951a5fd4a nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.148806536 +0000 UTC m=+51.964802335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxnkj" (UID: "9271574f-8b74-4de7-8bcf-f90951a5fd4a") : secret "networking-console-plugin-cert" not found Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148832 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148845 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7756d77bb6-lfwx6: secret "image-registry-tls" not found Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148863 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls podName:fe5633bb-4256-4685-943e-0e0ecbce30cd nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.148856454 +0000 UTC m=+51.964852254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls") pod "image-registry-5bffd989f9-r4bss" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd") : secret "image-registry-tls" not found Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148875 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.148869439 +0000 UTC m=+51.964865239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : configmap references non-existent config key: service-ca.crt Nov 02 19:47:51.148923 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.148912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh5c\" (UniqueName: \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.149225 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.148960 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls podName:2419a0c9-85c6-417d-9c5a-358e533246a9 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.148942974 +0000 UTC m=+51.964938779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls") pod "image-registry-7756d77bb6-lfwx6" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9") : secret "image-registry-tls" not found Nov 02 19:47:51.149225 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.149010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.149225 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.149060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:51.149225 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.149084 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.149225 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.149146 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:47:51.149225 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.149179 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:47:59.149170896 +0000 UTC m=+51.965166720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : secret "router-metrics-certs-default" not found Nov 02 19:47:51.149413 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.149300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.149695 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.149678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.206521 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.206493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh5c\" (UniqueName: \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.326912 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.326883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:47:51.383163 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.383113 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" podStartSLOduration=388.318138461 podStartE2EDuration="6m34.383100259s" totalImagesPullingTime="6.064961798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:51.381923374 +0000 UTC m=+44.197919190" watchObservedRunningTime="2026-11-02 19:47:51.383100259 +0000 UTC m=+44.199096080" Nov 02 19:47:51.451317 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.451265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:51.451482 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.451336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:51.451482 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.451401 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:47:51.451482 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.451417 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:47:51.451482 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.451468 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:52.451455799 +0000 UTC m=+45.267451612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : configmap "kube-rbac-proxy" not found Nov 02 19:47:51.451482 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.451483 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:52.451477391 +0000 UTC m=+45.267473204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : secret "insights-runtime-extractor-tls" not found Nov 02 19:47:51.531798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.531767 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4"] Nov 02 19:47:51.571636 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.571600 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt"] Nov 02 19:47:51.571766 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.571719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" Nov 02 19:47:51.589809 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.589782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" Nov 02 19:47:51.589809 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.589788 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4"] Nov 02 19:47:51.596303 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.596260 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt"] Nov 02 19:47:51.625024 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.624990 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2gbh4\"" Nov 02 19:47:51.625135 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.625112 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 19:47:51.625188 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.625172 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 19:47:51.654009 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.653985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdkl\" (UniqueName: \"kubernetes.io/projected/4b70b3c0-c849-4af0-904f-677aa9222a1e-kube-api-access-2xdkl\") pod \"migrator-66df5d496b-9gfq4\" (UID: \"4b70b3c0-c849-4af0-904f-677aa9222a1e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" Nov 02 19:47:51.654416 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.654396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjt2l\" (UniqueName: \"kubernetes.io/projected/426ee25e-f4e1-44da-b11f-191d799f8f27-kube-api-access-zjt2l\") pod \"migrator-66df5d496b-z2znt\" (UID: \"426ee25e-f4e1-44da-b11f-191d799f8f27\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" Nov 02 19:47:51.715297 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:51.715257 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674789e3_135f_4296_9de6_274e37588b04.slice/crio-28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990 WatchSource:0}: Error finding container 28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990: Status 404 returned error can't find the container with id 28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990 Nov 02 19:47:51.717209 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.717190 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gcdjm-447t7"] Nov 02 19:47:51.755061 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.755035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdkl\" (UniqueName: \"kubernetes.io/projected/4b70b3c0-c849-4af0-904f-677aa9222a1e-kube-api-access-2xdkl\") pod \"migrator-66df5d496b-9gfq4\" (UID: \"4b70b3c0-c849-4af0-904f-677aa9222a1e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" Nov 02 19:47:51.755189 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.755160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt2l\" (UniqueName: \"kubernetes.io/projected/426ee25e-f4e1-44da-b11f-191d799f8f27-kube-api-access-zjt2l\") pod \"migrator-66df5d496b-z2znt\" (UID: \"426ee25e-f4e1-44da-b11f-191d799f8f27\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" Nov 02 19:47:51.814300 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.814247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdkl\" (UniqueName: \"kubernetes.io/projected/4b70b3c0-c849-4af0-904f-677aa9222a1e-kube-api-access-2xdkl\") pod \"migrator-66df5d496b-9gfq4\" (UID: \"4b70b3c0-c849-4af0-904f-677aa9222a1e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" Nov 02 19:47:51.814300 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.814263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt2l\" (UniqueName: \"kubernetes.io/projected/426ee25e-f4e1-44da-b11f-191d799f8f27-kube-api-access-zjt2l\") pod \"migrator-66df5d496b-z2znt\" (UID: \"426ee25e-f4e1-44da-b11f-191d799f8f27\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" Nov 02 19:47:51.896470 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.896368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" Nov 02 19:47:51.915044 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.915023 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" Nov 02 19:47:51.926057 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.926021 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/1.log" Nov 02 19:47:51.926445 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.926422 2579 scope.go:122] "RemoveContainer" containerID="28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5" Nov 02 19:47:51.926687 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:51.926662 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2b245_openshift-console-operator(deaf3243-1eeb-4d94-bef9-524b67035ba2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" podUID="deaf3243-1eeb-4d94-bef9-524b67035ba2" Nov 02 19:47:51.927749 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.927722 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcdjm-447t7" event={"ID":"674789e3-135f-4296-9de6-274e37588b04","Type":"ContainerStarted","Data":"c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f"} Nov 02 19:47:51.927875 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:51.927771 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcdjm-447t7" event={"ID":"674789e3-135f-4296-9de6-274e37588b04","Type":"ContainerStarted","Data":"28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990"} Nov 02 19:47:52.153454 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.153385 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cwvdn"] Nov 02 19:47:52.163078 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.163027 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gcdjm-447t7" podStartSLOduration=2.163015742 podStartE2EDuration="2.163015742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:52.16192329 +0000 UTC m=+44.977919111" watchObservedRunningTime="2026-11-02 19:47:52.163015742 +0000 UTC m=+44.979011564" Nov 02 19:47:52.170849 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.170826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.211198 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.211169 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cwvdn"] Nov 02 19:47:52.224801 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.224772 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 19:47:52.224929 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.224838 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gkplk\"" Nov 02 19:47:52.224929 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.224852 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 19:47:52.224929 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.224869 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 19:47:52.224929 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.224857 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 19:47:52.261776 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.261749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.261908 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.261803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-key\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.261993 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.261974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6txz\" (UniqueName: \"kubernetes.io/projected/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-kube-api-access-r6txz\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.275841 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:52.275815 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b70b3c0_c849_4af0_904f_677aa9222a1e.slice/crio-f805b4c40af57e21c99de8bf14aa5092df02445c92ca37b97e7d6cd3bf4c1220 WatchSource:0}: Error finding container f805b4c40af57e21c99de8bf14aa5092df02445c92ca37b97e7d6cd3bf4c1220: Status 404 returned error can't find the container with id f805b4c40af57e21c99de8bf14aa5092df02445c92ca37b97e7d6cd3bf4c1220 Nov 02 19:47:52.278114 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.278091 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4"] Nov 02 19:47:52.300847 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:52.300826 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426ee25e_f4e1_44da_b11f_191d799f8f27.slice/crio-3c1913565262aaea69f7b1217d7f4e0bcb29f959991a42f7d4faafaa03fc2938 WatchSource:0}: Error finding container 3c1913565262aaea69f7b1217d7f4e0bcb29f959991a42f7d4faafaa03fc2938: Status 404 returned error can't find the container with id 3c1913565262aaea69f7b1217d7f4e0bcb29f959991a42f7d4faafaa03fc2938 Nov 02 19:47:52.303939 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.303919 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt"] Nov 02 19:47:52.362650 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.362624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.362822 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.362691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-key\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.362822 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.362791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6txz\" (UniqueName: \"kubernetes.io/projected/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-kube-api-access-r6txz\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.363322 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.363301 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.365414 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.365396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-key\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.420785 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.420696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6txz\" (UniqueName: \"kubernetes.io/projected/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-kube-api-access-r6txz\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.464146 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.464107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:52.464309 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.464189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:52.464309 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:52.464249 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:47:52.464409 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:52.464323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:54.464309524 +0000 UTC m=+47.280305338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : secret "insights-runtime-extractor-tls" not found Nov 02 19:47:52.464409 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:52.464372 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:47:52.464522 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:52.464433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:54.4644199 +0000 UTC m=+47.280415716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : configmap "kube-rbac-proxy" not found Nov 02 19:47:52.495263 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.495233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:47:52.867368 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:52.867339 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaf72f9_c2ab_4081_87f4_7747ffc180ad.slice/crio-0c8cbc0b3b6c6558a1e4256ba60bc23e37e9ad0f2f2af3538d52f899300c2d7b WatchSource:0}: Error finding container 0c8cbc0b3b6c6558a1e4256ba60bc23e37e9ad0f2f2af3538d52f899300c2d7b: Status 404 returned error can't find the container with id 0c8cbc0b3b6c6558a1e4256ba60bc23e37e9ad0f2f2af3538d52f899300c2d7b Nov 02 19:47:52.870632 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.870605 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cwvdn"] Nov 02 19:47:52.931762 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.931732 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" event={"ID":"dbaf72f9-c2ab-4081-87f4-7747ffc180ad","Type":"ContainerStarted","Data":"0c8cbc0b3b6c6558a1e4256ba60bc23e37e9ad0f2f2af3538d52f899300c2d7b"} Nov 02 19:47:52.933070 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.933047 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" event={"ID":"4b70b3c0-c849-4af0-904f-677aa9222a1e","Type":"ContainerStarted","Data":"f805b4c40af57e21c99de8bf14aa5092df02445c92ca37b97e7d6cd3bf4c1220"} Nov 02 19:47:52.934117 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:52.934089 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" event={"ID":"426ee25e-f4e1-44da-b11f-191d799f8f27","Type":"ContainerStarted","Data":"3c1913565262aaea69f7b1217d7f4e0bcb29f959991a42f7d4faafaa03fc2938"} Nov 02 19:47:53.587715 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:53.587471 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:53.588139 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:53.588122 2579 scope.go:122] "RemoveContainer" containerID="28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5" Nov 02 19:47:53.588529 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:53.588494 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:47:53.588660 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:53.588630 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2b245_openshift-console-operator(deaf3243-1eeb-4d94-bef9-524b67035ba2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" podUID="deaf3243-1eeb-4d94-bef9-524b67035ba2" Nov 02 19:47:53.940631 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:53.940601 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" event={"ID":"dbaf72f9-c2ab-4081-87f4-7747ffc180ad","Type":"ContainerStarted","Data":"f993aac75a1e9311c8594e864d30919e4407ddb16e102db9bc78492e30916f0a"} Nov 02 19:47:53.942040 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:53.942008 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" event={"ID":"4b70b3c0-c849-4af0-904f-677aa9222a1e","Type":"ContainerStarted","Data":"423ef47b8acc9f2aa955ecbe4b830a5f0b2325bf18fe7bf7211ee8a354fcb7ed"} Nov 02 19:47:53.942342 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:53.942327 2579 scope.go:122] "RemoveContainer" containerID="28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5" Nov 02 19:47:53.942617 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:53.942508 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2b245_openshift-console-operator(deaf3243-1eeb-4d94-bef9-524b67035ba2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" podUID="deaf3243-1eeb-4d94-bef9-524b67035ba2" Nov 02 19:47:54.202206 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:54.202120 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" podStartSLOduration=2.20210732 podStartE2EDuration="2.20210732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:54.083103251 +0000 UTC m=+46.899099076" watchObservedRunningTime="2026-11-02 19:47:54.20210732 +0000 UTC m=+47.018103142" Nov 02 19:47:54.486476 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:54.486384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:54.486476 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:54.486436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:54.486698 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:54.486563 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:47:54.486698 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:54.486590 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:47:54.486698 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:54.486644 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:58.486620801 +0000 UTC m=+51.302616618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : secret "insights-runtime-extractor-tls" not found Nov 02 19:47:54.486698 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:54.486665 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:47:58.486654698 +0000 UTC m=+51.302650503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : configmap "kube-rbac-proxy" not found Nov 02 19:47:54.947081 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:54.947044 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" event={"ID":"426ee25e-f4e1-44da-b11f-191d799f8f27","Type":"ContainerStarted","Data":"92306026845e021760d0162070385586b52ea0acdfa83e50cb1bd1d0d8508ea1"} Nov 02 19:47:55.063800 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:55.063746 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" podStartSLOduration=2.508662111 podStartE2EDuration="4.063731584s" totalImagesPullingTime="1.555069473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:54.201659664 +0000 UTC m=+47.017655485" watchObservedRunningTime="2026-11-02 19:47:55.063731584 +0000 UTC m=+47.879727444" Nov 02 19:47:55.064406 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:55.064379 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" podStartSLOduration=2.456128763 podStartE2EDuration="4.06437205s" totalImagesPullingTime="1.608243287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:47:55.063379401 +0000 UTC m=+47.879375223" watchObservedRunningTime="2026-11-02 19:47:55.06437205 +0000 UTC m=+47.880367897" Nov 02 19:47:55.860979 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:55.860951 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5bq8f_31464cb4-534f-41a2-bc06-99343f47a433/dns-node-resolver/0.log" Nov 02 19:47:57.061151 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:57.061123 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pcsq4_f40f9013-d584-4546-940b-82f31a20cd4f/node-ca/0.log" Nov 02 19:47:57.662929 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:57.662897 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dkb72_3b79f192-7b15-47d5-91b4-1772b3541ecf/serve-healthcheck-canary/0.log" Nov 02 19:47:57.862651 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:57.862624 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-9gfq4_4b70b3c0-c849-4af0-904f-677aa9222a1e/migrator/0.log" Nov 02 19:47:58.062766 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:58.062737 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-z2znt_426ee25e-f4e1-44da-b11f-191d799f8f27/migrator/0.log" Nov 02 19:47:58.266722 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:58.266693 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc_ff626920-a094-4782-aae1-ac6bc9d0b359/kube-storage-version-migrator-operator/0.log" Nov 02 19:47:58.524005 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:58.523970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:58.524146 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:58.524017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:47:58.524146 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:58.524089 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:47:58.524146 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:58.524124 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:47:58.524146 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:58.524140 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:48:06.524124567 +0000 UTC m=+59.340120386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : configmap "kube-rbac-proxy" not found Nov 02 19:47:58.524316 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:58.524180 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:48:06.524167367 +0000 UTC m=+59.340163204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : secret "insights-runtime-extractor-tls" not found Nov 02 19:47:59.130348 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.130311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:59.130793 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.130414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:59.130793 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.130451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:47:59.130793 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.130624 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:59.130793 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.130697 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls podName:97562254-d922-4415-9438-ae340b218942 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:15.130681052 +0000 UTC m=+67.946676871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kz9f7" (UID: "97562254-d922-4415-9438-ae340b218942") : secret "cluster-monitoring-operator-tls" not found Nov 02 19:47:59.133712 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.133684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:59.133822 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.133754 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:59.216433 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.216394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-27pq5" Nov 02 19:47:59.231685 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.231658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:59.231813 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.231711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:47:59.231813 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.231803 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:59.231921 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.231813 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 19:47:59.231921 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.231847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:47:59.231921 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.231888 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:15.231866197 +0000 UTC m=+68.047862012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : secret "router-metrics-certs-default" not found Nov 02 19:47:59.231921 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.231817 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bffd989f9-r4bss: secret "image-registry-tls" not found Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.231939 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.231934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") pod \"image-registry-7756d77bb6-lfwx6\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.231945 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls podName:fe5633bb-4256-4685-943e-0e0ecbce30cd nodeName:}" failed. No retries permitted until 2026-11-02 19:48:15.231931931 +0000 UTC m=+68.047927734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls") pod "image-registry-5bffd989f9-r4bss" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd") : secret "image-registry-tls" not found Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.232001 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert podName:9271574f-8b74-4de7-8bcf-f90951a5fd4a nodeName:}" failed. No retries permitted until 2026-11-02 19:48:15.231987286 +0000 UTC m=+68.047983089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-pxnkj" (UID: "9271574f-8b74-4de7-8bcf-f90951a5fd4a") : secret "networking-console-plugin-cert" not found Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.231999 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.232017 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7756d77bb6-lfwx6: secret "image-registry-tls" not found Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.232062 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls podName:2419a0c9-85c6-417d-9c5a-358e533246a9 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:15.232050285 +0000 UTC m=+68.048046099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls") pod "image-registry-7756d77bb6-lfwx6" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9") : secret "image-registry-tls" not found Nov 02 19:47:59.232171 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.232100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:47:59.232470 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:47:59.232256 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle podName:6f73e2f3-6f98-41bd-abec-b72d60ac2bb5 nodeName:}" failed. No retries permitted until 2026-11-02 19:48:15.232243373 +0000 UTC m=+68.048239174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle") pod "router-default-5f6668f8f7-2vjcr" (UID: "6f73e2f3-6f98-41bd-abec-b72d60ac2bb5") : configmap references non-existent config key: service-ca.crt Nov 02 19:47:59.265309 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.265258 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:47:59.579531 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:47:59.579485 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a620cde_dc26_4159_8e40_41fdd982d61f.slice/crio-583ca4a14a9c3c9c0e2e22b0e585b398b5bbf9212e7c55820e2e58006c6b0351 WatchSource:0}: Error finding container 583ca4a14a9c3c9c0e2e22b0e585b398b5bbf9212e7c55820e2e58006c6b0351: Status 404 returned error can't find the container with id 583ca4a14a9c3c9c0e2e22b0e585b398b5bbf9212e7c55820e2e58006c6b0351 Nov 02 19:47:59.582267 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.582245 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-27pq5"] Nov 02 19:47:59.636730 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.636701 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv"] Nov 02 19:47:59.965069 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.964977 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-27pq5" event={"ID":"1a620cde-dc26-4159-8e40-41fdd982d61f","Type":"ContainerStarted","Data":"583ca4a14a9c3c9c0e2e22b0e585b398b5bbf9212e7c55820e2e58006c6b0351"} Nov 02 19:47:59.966141 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:47:59.966116 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" event={"ID":"703df830-74cd-46d0-96ab-ab6b2b42c439","Type":"ContainerStarted","Data":"f5d921a9e6dafbdcab26570eed2c4559f056b94e0338c43bf0c18ce4d5b319a5"} Nov 02 19:48:02.644574 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.644540 2579 scope.go:122] "RemoveContainer" containerID="28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5" Nov 02 19:48:02.980989 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.980910 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:48:02.981351 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.981335 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/1.log" Nov 02 19:48:02.981396 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.981370 2579 generic.go:356] "Generic (PLEG): container finished" podID="deaf3243-1eeb-4d94-bef9-524b67035ba2" containerID="af3dc24e1e3b99eeaabf84378cdedff256741f25d6a0946fadd6d653d68e71c2" exitCode=255 Nov 02 19:48:02.981471 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.981454 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" event={"ID":"deaf3243-1eeb-4d94-bef9-524b67035ba2","Type":"ContainerDied","Data":"af3dc24e1e3b99eeaabf84378cdedff256741f25d6a0946fadd6d653d68e71c2"} Nov 02 19:48:02.981529 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.981516 2579 scope.go:122] "RemoveContainer" containerID="28c95b34ca8f3e228d1feb88a1540c03bea2abd154780df33fe87386c7e585b5" Nov 02 19:48:02.981922 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.981900 2579 scope.go:122] "RemoveContainer" containerID="af3dc24e1e3b99eeaabf84378cdedff256741f25d6a0946fadd6d653d68e71c2" Nov 02 19:48:02.982150 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:48:02.982127 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2b245_openshift-console-operator(deaf3243-1eeb-4d94-bef9-524b67035ba2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" podUID="deaf3243-1eeb-4d94-bef9-524b67035ba2" Nov 02 19:48:02.983401 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.983377 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" event={"ID":"703df830-74cd-46d0-96ab-ab6b2b42c439","Type":"ContainerStarted","Data":"013bd7ea0bdc74234e6c13682f38b78e74122abbf30c925a8f19d98d741c7021"} Nov 02 19:48:02.983478 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.983423 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" event={"ID":"703df830-74cd-46d0-96ab-ab6b2b42c439","Type":"ContainerStarted","Data":"2ed9e7502d665c2a96f77db905220ff798c5fc68e9fd0a5c09dfb46cd8f50255"} Nov 02 19:48:02.985048 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.985028 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-27pq5" event={"ID":"1a620cde-dc26-4159-8e40-41fdd982d61f","Type":"ContainerStarted","Data":"009bcb0be7647f9d25fd6d86dcb18abd38785e7e34f88aaac8969409b2f13697"} Nov 02 19:48:02.985130 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.985063 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-27pq5" event={"ID":"1a620cde-dc26-4159-8e40-41fdd982d61f","Type":"ContainerStarted","Data":"1eb60cf5c1aae5b379cecb0b34e32a8ee5102ce1b7f2db6bfa1488b297cd2bd1"} Nov 02 19:48:02.985182 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:02.985167 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:03.224788 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:03.224732 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-27pq5" podStartSLOduration=17.646099579 podStartE2EDuration="20.224717473s" totalImagesPullingTime="2.578617894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:03.222628297 +0000 UTC m=+56.038624119" watchObservedRunningTime="2026-11-02 19:48:03.224717473 +0000 UTC m=+56.040713296" Nov 02 19:48:03.339588 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:03.339531 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" podStartSLOduration=407.304230999 podStartE2EDuration="6m50.339516695s" totalImagesPullingTime="3.035285696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:03.338953284 +0000 UTC m=+56.154949106" watchObservedRunningTime="2026-11-02 19:48:03.339516695 +0000 UTC m=+56.155512518" Nov 02 19:48:03.587483 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:03.587448 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:03.587640 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:03.587512 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:03.989435 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:03.989406 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:48:03.989805 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:03.989745 2579 scope.go:122] "RemoveContainer" containerID="af3dc24e1e3b99eeaabf84378cdedff256741f25d6a0946fadd6d653d68e71c2" Nov 02 19:48:03.989929 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:48:03.989910 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-2b245_openshift-console-operator(deaf3243-1eeb-4d94-bef9-524b67035ba2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" podUID="deaf3243-1eeb-4d94-bef9-524b67035ba2" Nov 02 19:48:06.600489 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:06.600448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:06.600853 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:06.600502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:06.600853 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:48:06.600619 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 19:48:06.600853 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:48:06.600646 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 19:48:06.600853 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:48:06.600702 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:48:22.600687079 +0000 UTC m=+75.416682892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : secret "insights-runtime-extractor-tls" not found Nov 02 19:48:06.600853 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:48:06.600719 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm podName:7bd90b6c-66f9-4ea8-8f73-2981acce43db nodeName:}" failed. No retries permitted until 2026-11-02 19:48:22.600712582 +0000 UTC m=+75.416708422 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x6hpx" (UID: "7bd90b6c-66f9-4ea8-8f73-2981acce43db") : configmap "kube-rbac-proxy" not found Nov 02 19:48:07.854435 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:07.854406 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:12.155246 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.155214 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7756d77bb6-lfwx6"] Nov 02 19:48:12.155703 ip-10-0-134-13 kubenswrapper[2579]: E1102 19:48:12.155487 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" podUID="2419a0c9-85c6-417d-9c5a-358e533246a9" Nov 02 19:48:12.186510 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.186481 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d458f586f-ndk5f"] Nov 02 19:48:12.191042 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.191024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253014 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.252976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-bound-sa-token\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253180 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.253098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h788\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-kube-api-access-8h788\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253180 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.253128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-image-registry-private-configuration\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253319 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.253244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-registry-certificates\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253319 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.253306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-registry-tls\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253411 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.253334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a63760e-f41b-4958-83a5-e4944090dccb-ca-trust-extracted\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253459 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.253418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-installation-pull-secrets\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.253494 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.253473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-trusted-ca\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.353933 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.353896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-bound-sa-token\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.353977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h788\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-kube-api-access-8h788\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.354002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-image-registry-private-configuration\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.354041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-registry-certificates\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.354068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-registry-tls\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354112 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.354092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a63760e-f41b-4958-83a5-e4944090dccb-ca-trust-extracted\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354414 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.354254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-installation-pull-secrets\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354414 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.354357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-trusted-ca\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.354577 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.354558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a63760e-f41b-4958-83a5-e4944090dccb-ca-trust-extracted\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.355056 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.355030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-registry-certificates\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.355192 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.355173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-trusted-ca\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.356795 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.356757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-image-registry-private-configuration\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.356918 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.356827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-registry-tls\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.356918 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.356851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-installation-pull-secrets\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.400257 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.400226 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d458f586f-ndk5f"] Nov 02 19:48:12.412511 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.412452 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-bound-sa-token\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.412619 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.412572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h788\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-kube-api-access-8h788\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.555477 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.555446 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5cssh\"" Nov 02 19:48:12.563618 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.563595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:12.992296 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:12.992257 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:13.019541 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.019511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:48:13.027326 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.027297 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d458f586f-ndk5f"] Nov 02 19:48:13.032200 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.032179 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:48:13.056057 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:13.056033 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a63760e_f41b_4958_83a5_e4944090dccb.slice/crio-97153f35a89bd7557d75c9227d324d3dec78630194e4a8c6947fe2dcfc9c7cb8 WatchSource:0}: Error finding container 97153f35a89bd7557d75c9227d324d3dec78630194e4a8c6947fe2dcfc9c7cb8: Status 404 returned error can't find the container with id 97153f35a89bd7557d75c9227d324d3dec78630194e4a8c6947fe2dcfc9c7cb8 Nov 02 19:48:13.163147 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-bound-sa-token\") pod \"2419a0c9-85c6-417d-9c5a-358e533246a9\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " Nov 02 19:48:13.163602 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163194 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-certificates\") pod \"2419a0c9-85c6-417d-9c5a-358e533246a9\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " Nov 02 19:48:13.163602 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-trusted-ca\") pod \"2419a0c9-85c6-417d-9c5a-358e533246a9\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " Nov 02 19:48:13.163602 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163317 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-image-registry-private-configuration\") pod \"2419a0c9-85c6-417d-9c5a-358e533246a9\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " Nov 02 19:48:13.163602 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163358 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-kube-api-access-66qrm\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-kube-api-access-66qrm\") pod \"2419a0c9-85c6-417d-9c5a-358e533246a9\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " Nov 02 19:48:13.163602 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163400 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-installation-pull-secrets\") pod \"2419a0c9-85c6-417d-9c5a-358e533246a9\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " Nov 02 19:48:13.163602 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163488 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2419a0c9-85c6-417d-9c5a-358e533246a9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2419a0c9-85c6-417d-9c5a-358e533246a9-ca-trust-extracted\") pod \"2419a0c9-85c6-417d-9c5a-358e533246a9\" (UID: \"2419a0c9-85c6-417d-9c5a-358e533246a9\") " Nov 02 19:48:13.163963 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163827 2579 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-trusted-ca" pod "2419a0c9-85c6-417d-9c5a-358e533246a9" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:13.163963 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.163909 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-trusted-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.164084 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.164000 2579 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2419a0c9-85c6-417d-9c5a-358e533246a9-ca-trust-extracted" pod "2419a0c9-85c6-417d-9c5a-358e533246a9" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:48:13.164160 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.164140 2579 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-certificates" pod "2419a0c9-85c6-417d-9c5a-358e533246a9" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:13.165941 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.165882 2579 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-bound-sa-token" pod "2419a0c9-85c6-417d-9c5a-358e533246a9" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:13.166231 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.166053 2579 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-installation-pull-secrets" pod "2419a0c9-85c6-417d-9c5a-358e533246a9" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:13.166231 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.166095 2579 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-image-registry-private-configuration" pod "2419a0c9-85c6-417d-9c5a-358e533246a9" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:13.166231 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.166187 2579 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-kube-api-access-66qrm" pod "2419a0c9-85c6-417d-9c5a-358e533246a9" (UID: "2419a0c9-85c6-417d-9c5a-358e533246a9"). InnerVolumeSpecName "kube-api-access-66qrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:13.265145 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.265105 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2419a0c9-85c6-417d-9c5a-358e533246a9-ca-trust-extracted\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.265145 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.265134 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-bound-sa-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.265145 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.265146 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-certificates\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.265145 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.265157 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-image-registry-private-configuration\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.265520 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.265166 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66qrm\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-kube-api-access-66qrm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.265520 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.265175 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2419a0c9-85c6-417d-9c5a-358e533246a9-installation-pull-secrets\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:13.467798 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.467757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:48:13.468006 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.467924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:13.520525 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.520451 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:48:13.520525 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.520480 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:48:13.530976 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.530951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:48:13.531094 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.530978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:13.569034 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.569005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:13.571529 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.571481 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:13.854803 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.854718 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\"" Nov 02 19:48:13.862849 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.862828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:48:13.867679 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.867662 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\"" Nov 02 19:48:13.875927 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.875903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:13.906373 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.906339 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\"" Nov 02 19:48:13.912474 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:13.912417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:14.024496 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.024462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7756d77bb6-lfwx6" Nov 02 19:48:14.024663 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.024501 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" event={"ID":"0a63760e-f41b-4958-83a5-e4944090dccb","Type":"ContainerStarted","Data":"9a15bc014ea6cd7603a49ff7bb1db1ece621c8099f458b5f9e149ca0ac0bdcfe"} Nov 02 19:48:14.024663 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.024652 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:14.024764 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.024688 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" event={"ID":"0a63760e-f41b-4958-83a5-e4944090dccb","Type":"ContainerStarted","Data":"97153f35a89bd7557d75c9227d324d3dec78630194e4a8c6947fe2dcfc9c7cb8"} Nov 02 19:48:14.144729 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.144643 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" podStartSLOduration=2.144628406 podStartE2EDuration="2.144628406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:14.143686487 +0000 UTC m=+66.959682308" watchObservedRunningTime="2026-11-02 19:48:14.144628406 +0000 UTC m=+66.960624228" Nov 02 19:48:14.244542 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:14.244513 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea63025_526f_4fd8_b303_bc115c1226dd.slice/crio-881f780a8a58bd748681d528700d9fec13aee1c088482468c4a2b75aa5a23b85 WatchSource:0}: Error finding container 881f780a8a58bd748681d528700d9fec13aee1c088482468c4a2b75aa5a23b85: Status 404 returned error can't find the container with id 881f780a8a58bd748681d528700d9fec13aee1c088482468c4a2b75aa5a23b85 Nov 02 19:48:14.247730 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.247706 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nt26r"] Nov 02 19:48:14.290612 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:14.290587 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeceae17b_5b0b_45ce_87cf_9d720583c247.slice/crio-f8fc59e78c7193eab0c62117404d42facf071020aef535cc2cdf95e1da9bf867 WatchSource:0}: Error finding container f8fc59e78c7193eab0c62117404d42facf071020aef535cc2cdf95e1da9bf867: Status 404 returned error can't find the container with id f8fc59e78c7193eab0c62117404d42facf071020aef535cc2cdf95e1da9bf867 Nov 02 19:48:14.291459 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:14.291435 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505ade63_4db5_4709_9e6a_cdaed488b662.slice/crio-5b309c160687ca5349bdb92e83e4a4e7f232131269379c381c3bfc54c1f89cc0 WatchSource:0}: Error finding container 5b309c160687ca5349bdb92e83e4a4e7f232131269379c381c3bfc54c1f89cc0: Status 404 returned error can't find the container with id 5b309c160687ca5349bdb92e83e4a4e7f232131269379c381c3bfc54c1f89cc0 Nov 02 19:48:14.292214 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.292193 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j7kp4"] Nov 02 19:48:14.293847 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.293816 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mpxzs"] Nov 02 19:48:14.320710 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.320686 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7756d77bb6-lfwx6"] Nov 02 19:48:14.325055 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.325034 2579 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7756d77bb6-lfwx6"] Nov 02 19:48:14.479171 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:14.479094 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2419a0c9-85c6-417d-9c5a-358e533246a9-registry-tls\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:15.028411 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.028373 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j7kp4" event={"ID":"eceae17b-5b0b-45ce-87cf-9d720583c247","Type":"ContainerStarted","Data":"f8fc59e78c7193eab0c62117404d42facf071020aef535cc2cdf95e1da9bf867"} Nov 02 19:48:15.029490 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.029462 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nt26r" event={"ID":"fea63025-526f-4fd8-b303-bc115c1226dd","Type":"ContainerStarted","Data":"881f780a8a58bd748681d528700d9fec13aee1c088482468c4a2b75aa5a23b85"} Nov 02 19:48:15.030913 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.030890 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mpxzs" event={"ID":"505ade63-4db5-4709-9e6a-cdaed488b662","Type":"ContainerStarted","Data":"5c70ac3030ed6891e0bff0d31f727632e735b9cb0a508b39ae2503e7675b0df9"} Nov 02 19:48:15.030992 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.030932 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mpxzs" event={"ID":"505ade63-4db5-4709-9e6a-cdaed488b662","Type":"ContainerStarted","Data":"5b309c160687ca5349bdb92e83e4a4e7f232131269379c381c3bfc54c1f89cc0"} Nov 02 19:48:15.145904 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.145843 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mpxzs" podStartSLOduration=67.145826311 podStartE2EDuration="1m7.145826311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:15.145302653 +0000 UTC m=+67.961298471" watchObservedRunningTime="2026-11-02 19:48:15.145826311 +0000 UTC m=+67.961822132" Nov 02 19:48:15.185324 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.185266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:15.188138 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.188111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:15.286589 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.286502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:15.286589 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.286578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:15.287059 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.286609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:48:15.287059 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.286668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:15.287472 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.287447 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:15.289359 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.289338 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:15.289479 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.289410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:15.289755 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.289732 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:48:15.380381 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.380350 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t6dtl\"" Nov 02 19:48:15.388517 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.388487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:15.494364 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.494328 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mg6c7\"" Nov 02 19:48:15.502576 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.502548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:15.532140 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.532106 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:15.560300 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.560221 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-869qp\"" Nov 02 19:48:15.567634 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.567451 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:48:15.653008 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.652975 2579 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2419a0c9-85c6-417d-9c5a-358e533246a9" path="/var/lib/kubelet/pods/2419a0c9-85c6-417d-9c5a-358e533246a9/volumes" Nov 02 19:48:15.802415 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:15.802379 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97562254_d922_4415_9438_ae340b218942.slice/crio-31d628e29b74262ad921ff1bc39a244ca7786a157e0f80b051a5b1e79d403084 WatchSource:0}: Error finding container 31d628e29b74262ad921ff1bc39a244ca7786a157e0f80b051a5b1e79d403084: Status 404 returned error can't find the container with id 31d628e29b74262ad921ff1bc39a244ca7786a157e0f80b051a5b1e79d403084 Nov 02 19:48:15.805084 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.805057 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7"] Nov 02 19:48:15.911458 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:15.911408 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f73e2f3_6f98_41bd_abec_b72d60ac2bb5.slice/crio-58f1a311a9c9dc3d9144f6f549b585a3b17439db8b864c3ea821950fd89a4d5c WatchSource:0}: Error finding container 58f1a311a9c9dc3d9144f6f549b585a3b17439db8b864c3ea821950fd89a4d5c: Status 404 returned error can't find the container with id 58f1a311a9c9dc3d9144f6f549b585a3b17439db8b864c3ea821950fd89a4d5c Nov 02 19:48:15.911676 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.911654 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f6668f8f7-2vjcr"] Nov 02 19:48:15.953863 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:15.953822 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5633bb_4256_4685_943e_0e0ecbce30cd.slice/crio-fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b WatchSource:0}: Error finding container fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b: Status 404 returned error can't find the container with id fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b Nov 02 19:48:15.956177 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.956151 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bffd989f9-r4bss"] Nov 02 19:48:15.961159 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:15.961055 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-pxnkj"] Nov 02 19:48:15.967876 ip-10-0-134-13 kubenswrapper[2579]: W1102 19:48:15.966602 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9271574f_8b74_4de7_8bcf_f90951a5fd4a.slice/crio-85b786d7f7f6161e291510ecae10ea6462c9dc23fe272a282ad908013cccfb57 WatchSource:0}: Error finding container 85b786d7f7f6161e291510ecae10ea6462c9dc23fe272a282ad908013cccfb57: Status 404 returned error can't find the container with id 85b786d7f7f6161e291510ecae10ea6462c9dc23fe272a282ad908013cccfb57 Nov 02 19:48:16.035579 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.035540 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" event={"ID":"9271574f-8b74-4de7-8bcf-f90951a5fd4a","Type":"ContainerStarted","Data":"85b786d7f7f6161e291510ecae10ea6462c9dc23fe272a282ad908013cccfb57"} Nov 02 19:48:16.036992 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.036822 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" event={"ID":"97562254-d922-4415-9438-ae340b218942","Type":"ContainerStarted","Data":"31d628e29b74262ad921ff1bc39a244ca7786a157e0f80b051a5b1e79d403084"} Nov 02 19:48:16.038461 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.038437 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" event={"ID":"fe5633bb-4256-4685-943e-0e0ecbce30cd","Type":"ContainerStarted","Data":"6c79c2ff2528f0aa8829f2452b1a4fd8d52c06218e22fccbf44a019ba129d502"} Nov 02 19:48:16.038790 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.038478 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" event={"ID":"fe5633bb-4256-4685-943e-0e0ecbce30cd","Type":"ContainerStarted","Data":"fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b"} Nov 02 19:48:16.038790 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.038570 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:16.040126 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.040105 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" event={"ID":"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5","Type":"ContainerStarted","Data":"8a993085050de26e91a9d4c10d7457ce6564a00bc7b1c00642f11be523e24caa"} Nov 02 19:48:16.040208 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.040139 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" event={"ID":"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5","Type":"ContainerStarted","Data":"58f1a311a9c9dc3d9144f6f549b585a3b17439db8b864c3ea821950fd89a4d5c"} Nov 02 19:48:16.040409 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.040394 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:16.160087 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.160022 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" podStartSLOduration=59.160001771 podStartE2EDuration="59.160001771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:16.158341064 +0000 UTC m=+68.974336885" watchObservedRunningTime="2026-11-02 19:48:16.160001771 +0000 UTC m=+68.975997593" Nov 02 19:48:16.277110 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.277059 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" podStartSLOduration=421.277040291 podStartE2EDuration="7m1.277040291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:16.276466484 +0000 UTC m=+69.092462305" watchObservedRunningTime="2026-11-02 19:48:16.277040291 +0000 UTC m=+69.093036112" Nov 02 19:48:16.503722 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.503688 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:16.506932 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:16.506904 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:17.047248 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:17.046927 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nt26r" event={"ID":"fea63025-526f-4fd8-b303-bc115c1226dd","Type":"ContainerStarted","Data":"70037a09de289b9674411f9590b58c23ae2a143f3e1f6282f8b07d7f028284cd"} Nov 02 19:48:17.047458 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:17.047254 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:17.047458 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:17.047313 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nt26r" event={"ID":"fea63025-526f-4fd8-b303-bc115c1226dd","Type":"ContainerStarted","Data":"70be560c435b0222e690db16637d2cbf14b885b750b0da36229165ab903c67ef"} Nov 02 19:48:17.048781 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:17.048732 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:17.164672 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:17.164449 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nt26r" podStartSLOduration=67.372449344 podStartE2EDuration="1m9.164427803s" totalImagesPullingTime="1.791978459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:17.161728812 +0000 UTC m=+69.977724634" watchObservedRunningTime="2026-11-02 19:48:17.164427803 +0000 UTC m=+69.980423626" Nov 02 19:48:18.053066 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:18.053025 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" event={"ID":"9271574f-8b74-4de7-8bcf-f90951a5fd4a","Type":"ContainerStarted","Data":"2bca12a529d2265f248173dbec6396e638781f211d8ce610cb9c30c24c76312a"} Nov 02 19:48:18.172064 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:18.172014 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" podStartSLOduration=406.489873916 podStartE2EDuration="6m48.172001847s" totalImagesPullingTime="1.682127931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:18.171216103 +0000 UTC m=+70.987211924" watchObservedRunningTime="2026-11-02 19:48:18.172001847 +0000 UTC m=+70.987997669" Nov 02 19:48:19.057822 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:19.057787 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" event={"ID":"97562254-d922-4415-9438-ae340b218942","Type":"ContainerStarted","Data":"e36de8402f5d5c28894b1fb4319f082e793fc7e1cdc256d5c993368b160eaff6"} Nov 02 19:48:19.174306 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:19.174231 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" podStartSLOduration=425.260541421 podStartE2EDuration="7m8.174213362s" totalImagesPullingTime="2.913671941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:19.173159869 +0000 UTC m=+71.989155692" watchObservedRunningTime="2026-11-02 19:48:19.174213362 +0000 UTC m=+71.990209232" Nov 02 19:48:21.416307 ip-10-0-134-13 kubenswrapper[2579]: I1102 19:48:21.416131 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:48:21.416334 ip-10-0-134-13 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 19:48:21.424892 ip-10-0-134-13 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 19:48:21.425055 ip-10-0-134-13 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 19:48:21.425231 ip-10-0-134-13 systemd[1]: kubelet.service: Consumed 4.662s CPU time. Nov 02 19:48:21.441241 ip-10-0-134-13 systemd[1]: Starting Kubernetes Kubelet... Nov 02 19:48:21.531104 ip-10-0-134-13 kubenswrapper[5800]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:48:21.531104 ip-10-0-134-13 kubenswrapper[5800]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 19:48:21.531104 ip-10-0-134-13 kubenswrapper[5800]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:48:21.531104 ip-10-0-134-13 kubenswrapper[5800]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 19:48:21.533151 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533057 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.533151 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533151 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533154 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533158 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533160 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533164 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533166 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533169 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533172 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533174 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533178 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533181 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533184 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533186 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533189 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533192 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533195 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533197 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533200 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533203 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533206 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.533221 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533208 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533211 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533221 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533225 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533228 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533231 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533233 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533236 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533239 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533242 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533245 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533247 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533250 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533253 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533255 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533258 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533261 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533263 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533266 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533268 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.533758 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533271 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533274 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533290 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533294 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533296 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533299 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533302 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533305 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533309 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533311 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533314 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533317 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533319 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533322 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533325 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533358 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533362 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533365 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533369 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.534246 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533373 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533378 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533380 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533383 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533386 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533388 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533391 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533394 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533396 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533399 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533403 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533407 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533410 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533413 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533416 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533418 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533421 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533424 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533427 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.534763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533430 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533433 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533436 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533439 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533441 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533444 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533447 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533449 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533452 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533455 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533457 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533460 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533462 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533465 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533468 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533471 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533473 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533476 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533478 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533481 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.535236 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533484 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533486 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533489 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533491 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533494 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533497 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.533500 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534307 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534316 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534319 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534322 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534325 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534328 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534331 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534333 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534336 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534339 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534342 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534345 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.535748 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534348 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534350 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534353 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534356 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534359 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534362 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534364 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534367 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534370 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534373 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534375 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534378 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534380 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534383 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534386 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534388 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534391 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534394 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534397 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.536202 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534399 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534402 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534405 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534414 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534416 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534419 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534422 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534424 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534427 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534429 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534432 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534435 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534437 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534440 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534443 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534445 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534448 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534451 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534453 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534456 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.536688 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534458 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534461 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534464 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534466 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534469 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534471 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534474 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534477 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534480 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534482 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534485 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534489 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534493 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534496 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534499 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534504 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534506 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534509 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534511 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534514 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.537189 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534517 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534519 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534522 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534524 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534527 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534530 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534532 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534537 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534540 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534543 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534546 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534549 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534553 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534557 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534560 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534563 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534566 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534568 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534571 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.537694 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534573 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534576 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534579 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534581 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534584 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534587 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534590 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534593 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534597 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534600 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534602 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534605 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534607 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534610 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534613 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.534615 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535143 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535149 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535152 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.538153 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535154 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535158 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535160 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535163 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535166 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535170 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535174 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535177 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535179 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535182 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535185 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535187 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535190 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535192 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535195 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535197 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535200 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535202 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535205 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.538643 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535209 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535212 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535217 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535220 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535223 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535225 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535228 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535231 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535233 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535236 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535238 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535241 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535244 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535247 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535250 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535252 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535255 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535258 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535261 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.539139 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535263 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535266 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535269 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535271 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535274 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535290 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535293 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535296 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535299 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535301 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535304 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535306 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535309 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535312 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535314 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535317 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535320 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535323 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535325 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.539663 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535329 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535331 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535334 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535337 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535339 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535342 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535344 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535347 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535349 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535352 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535355 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535357 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535360 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535363 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535365 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535368 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535371 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535373 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535376 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535379 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.540154 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535381 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535384 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535386 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535389 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535391 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535394 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535397 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535399 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535402 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535405 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535408 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535410 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535412 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535415 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535418 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535421 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535423 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535426 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535428 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535431 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.540664 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535436 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535440 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535444 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535449 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535452 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.535458 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535566 5800 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535574 5800 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535582 5800 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535590 5800 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535596 5800 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535600 5800 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535605 5800 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535609 5800 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535613 5800 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535616 5800 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535620 5800 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535623 5800 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535626 5800 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535629 5800 flags.go:64] FLAG: --cgroup-root="" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535632 5800 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535635 5800 flags.go:64] FLAG: --client-ca-file="" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535638 5800 flags.go:64] FLAG: --cloud-provider="external" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535641 5800 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 19:48:21.541146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535646 5800 flags.go:64] FLAG: --cluster-domain="" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535649 5800 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535652 5800 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535655 5800 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535658 5800 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535662 5800 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535665 5800 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535668 5800 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535676 5800 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535679 5800 flags.go:64] FLAG: --contention-profiling="false" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535682 5800 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535685 5800 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535688 5800 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535692 5800 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535696 5800 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535700 5800 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535703 5800 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535706 5800 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535709 5800 flags.go:64] FLAG: --enable-server="true" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535714 5800 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535718 5800 flags.go:64] FLAG: --event-burst="100" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535721 5800 flags.go:64] FLAG: --event-qps="50" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535724 5800 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535727 5800 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 19:48:21.541736 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535731 5800 flags.go:64] FLAG: --eviction-hard="" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535735 5800 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535738 5800 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535741 5800 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535744 5800 flags.go:64] FLAG: --eviction-soft="" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535747 5800 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535750 5800 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535753 5800 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535757 5800 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535760 5800 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535763 5800 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535766 5800 flags.go:64] FLAG: --feature-gates="" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535770 5800 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535773 5800 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535776 5800 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535780 5800 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535784 5800 flags.go:64] FLAG: --healthz-port="10248" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535787 5800 flags.go:64] FLAG: --help="false" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535790 5800 flags.go:64] FLAG: --hostname-override="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535794 5800 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535797 5800 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535800 5800 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535803 5800 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 19:48:21.542344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535806 5800 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535810 5800 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535813 5800 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535816 5800 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535820 5800 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535823 5800 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535826 5800 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535830 5800 flags.go:64] FLAG: --kube-reserved="" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535833 5800 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535835 5800 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535839 5800 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535841 5800 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535844 5800 flags.go:64] FLAG: --lock-file="" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535847 5800 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535850 5800 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535853 5800 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535858 5800 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535861 5800 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535865 5800 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535868 5800 flags.go:64] FLAG: --logging-format="text" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535871 5800 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535874 5800 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535877 5800 flags.go:64] FLAG: --manifest-url="" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535880 5800 flags.go:64] FLAG: --manifest-url-header="" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535884 5800 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 19:48:21.542897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535889 5800 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535893 5800 flags.go:64] FLAG: --max-pods="110" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535895 5800 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535899 5800 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535902 5800 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535905 5800 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535908 5800 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535910 5800 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535914 5800 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535920 5800 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535924 5800 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535928 5800 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535932 5800 flags.go:64] FLAG: --pod-cidr="" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535935 5800 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535938 5800 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535941 5800 flags.go:64] FLAG: --pods-per-core="0" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535944 5800 flags.go:64] FLAG: --port="10250" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535947 5800 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535950 5800 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ed9d6826d9bb117f" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535954 5800 flags.go:64] FLAG: --qos-reserved="" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535957 5800 flags.go:64] FLAG: --read-only-port="10255" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535960 5800 flags.go:64] FLAG: --register-node="true" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535963 5800 flags.go:64] FLAG: --register-with-taints="" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535969 5800 flags.go:64] FLAG: --registry-burst="10" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535972 5800 flags.go:64] FLAG: --registry-qps="5" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535975 5800 flags.go:64] FLAG: --reserved-cpus="" Nov 02 19:48:21.543545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535978 5800 flags.go:64] FLAG: --reserved-memory="" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535982 5800 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535986 5800 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535989 5800 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535992 5800 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535995 5800 flags.go:64] FLAG: --runonce="false" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.535999 5800 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536002 5800 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536005 5800 flags.go:64] FLAG: --seccomp-default="false" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536008 5800 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536011 5800 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536014 5800 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536017 5800 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536021 5800 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536024 5800 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536027 5800 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536030 5800 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536035 5800 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536038 5800 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536041 5800 flags.go:64] FLAG: --system-cgroups="" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536044 5800 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536049 5800 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536052 5800 flags.go:64] FLAG: --tls-cert-file="" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536056 5800 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536060 5800 flags.go:64] FLAG: --tls-min-version="" Nov 02 19:48:21.544412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536063 5800 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536066 5800 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536069 5800 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536072 5800 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536075 5800 flags.go:64] FLAG: --v="2" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536081 5800 flags.go:64] FLAG: --version="false" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536087 5800 flags.go:64] FLAG: --vmodule="" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536093 5800 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536096 5800 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536225 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536228 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536231 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536234 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536238 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536241 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536243 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536246 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536249 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536251 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536254 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536256 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.545405 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536259 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536261 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536264 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536269 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536272 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536275 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536289 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536292 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536294 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536297 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536300 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536303 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536306 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536308 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536311 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536314 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536316 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536319 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536322 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536324 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.545963 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536327 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536329 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536333 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536337 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536342 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536344 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536347 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536350 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536353 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536355 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536358 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536361 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536363 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536366 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536368 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536372 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536376 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536378 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536381 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.546484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536384 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536386 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536389 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536391 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536394 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536397 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536399 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536402 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536405 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536407 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536410 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536412 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536415 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536418 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536420 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536423 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536425 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536430 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536433 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536436 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.546995 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536438 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536441 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536445 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536448 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536451 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536454 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536457 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536459 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536463 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536466 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536469 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536472 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536474 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536477 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536479 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536482 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536485 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536487 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536490 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.547640 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536492 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536495 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536497 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536500 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536503 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536505 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536508 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536511 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536514 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536516 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536520 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536523 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536526 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536528 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536531 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.548126 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536533 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.536560 5800 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536595 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536600 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536603 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536606 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536610 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536613 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536616 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536619 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536622 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536625 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536628 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536631 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536635 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536639 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.548545 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536642 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536645 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536648 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536651 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536654 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536656 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536659 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536662 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536664 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536667 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536670 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536672 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536676 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536679 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536681 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536684 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536687 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536690 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536693 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536695 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.548991 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536698 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536701 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536705 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536708 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536710 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536713 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536717 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536719 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536722 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536725 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536727 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536730 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536733 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536735 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536738 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536741 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536744 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536748 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536752 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.550099 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536755 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536758 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536760 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536763 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536766 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536770 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536772 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536775 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536777 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536780 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536783 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536785 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536788 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536791 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536793 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536797 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536800 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536802 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536805 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.550846 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536808 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536811 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536814 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536817 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536819 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536823 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536826 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536828 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536831 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536833 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536836 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536839 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536841 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536844 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536846 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536849 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536852 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536854 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536858 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.551616 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536861 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536864 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536866 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536869 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536871 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536874 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536877 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536879 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536882 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536886 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536888 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536891 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536893 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536896 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.536899 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.541092 5800 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.541108 5800 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.541125 5800 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.541130 5800 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541195 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.552498 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541201 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541205 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541208 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541211 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541214 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541217 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541220 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541223 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541225 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541228 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541231 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541233 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541236 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541239 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541242 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541245 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541248 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541250 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541253 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541256 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.553129 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541259 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541261 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541264 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541266 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541269 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541272 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541275 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541294 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541299 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541302 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541304 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541307 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541310 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541312 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541315 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541317 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541320 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541323 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541325 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541328 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.553733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541330 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541333 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541335 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541338 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541341 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541344 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541347 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541349 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541352 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541355 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541358 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541361 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541363 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541366 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541369 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541371 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541374 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541377 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541380 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.554303 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541382 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541391 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541394 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541397 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541399 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541402 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541404 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541407 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541410 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541412 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541415 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541418 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541420 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541423 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541426 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541428 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541431 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541433 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541436 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.554796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541438 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541441 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541443 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541446 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541449 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541453 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541457 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541460 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541463 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541466 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541468 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541471 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541474 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541476 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541479 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541488 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541492 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541495 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541498 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.555259 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541501 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541503 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541506 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541509 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541512 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541514 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541517 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541519 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.541549 5800 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541755 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541760 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541763 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541765 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541768 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541770 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541773 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.555763 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541776 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541778 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541781 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541783 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541786 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541788 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541791 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541793 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541796 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541799 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541803 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541807 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541810 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541819 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541822 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541824 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541827 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541830 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541832 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.556164 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541835 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541838 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541840 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541843 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541845 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541848 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541850 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541853 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541855 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541858 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541860 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541863 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541865 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541868 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541870 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541873 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541876 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541878 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541881 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.556682 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541883 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541886 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541888 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541891 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541894 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541896 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541899 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541901 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541909 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541912 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541914 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541917 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541919 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541923 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541926 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541928 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541931 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541934 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541936 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541939 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.557185 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541941 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541944 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541946 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541949 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541951 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541954 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541956 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541960 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541963 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541966 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541968 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541971 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541973 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541976 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541978 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541984 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541986 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541989 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541991 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541994 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.557786 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.541996 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542005 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542008 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542010 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542013 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542016 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542018 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542021 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542024 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542026 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542029 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542031 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542034 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542036 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542039 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542041 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542044 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542047 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542049 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.558298 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542052 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542054 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.542076 5800 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542131 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542136 5800 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542139 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542143 5800 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542146 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542151 5800 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542153 5800 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542156 5800 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542159 5800 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542161 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542164 5800 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542167 5800 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542175 5800 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 19:48:21.558801 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542178 5800 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542181 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542183 5800 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542186 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542188 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542191 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542193 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542196 5800 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542199 5800 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542201 5800 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542204 5800 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542206 5800 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542209 5800 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542211 5800 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542214 5800 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542216 5800 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542219 5800 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542221 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542224 5800 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 19:48:21.559191 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542226 5800 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542229 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542231 5800 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542234 5800 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542237 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542240 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542243 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542246 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542249 5800 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542252 5800 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542254 5800 feature_gate.go:386] unrecognized feature gate: Example Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542257 5800 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542259 5800 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542267 5800 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542270 5800 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542273 5800 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542275 5800 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542289 5800 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542291 5800 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542294 5800 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 19:48:21.559733 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542296 5800 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542302 5800 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542306 5800 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542309 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542313 5800 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542316 5800 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542318 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542321 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542323 5800 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542326 5800 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542329 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542331 5800 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542334 5800 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542336 5800 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542339 5800 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542342 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542345 5800 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542349 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542352 5800 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 19:48:21.560251 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542355 5800 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542358 5800 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542361 5800 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542363 5800 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542366 5800 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542369 5800 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542371 5800 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542379 5800 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542382 5800 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542385 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542387 5800 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542390 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542393 5800 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542395 5800 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542399 5800 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542401 5800 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542404 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542406 5800 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542409 5800 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 19:48:21.560741 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542411 5800 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542414 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542416 5800 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542419 5800 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542421 5800 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542424 5800 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542426 5800 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542429 5800 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542431 5800 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542434 5800 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542436 5800 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542440 5800 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542443 5800 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542445 5800 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542450 5800 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:21.542454 5800 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.542504 5800 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.543491 5800 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.543562 5800 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 19:48:21.561205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.544054 5800 server.go:1014] "Starting client certificate rotation" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.544859 5800 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.544925 5800 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 19:42:07 +0000 UTC" deadline="2028-07-21 09:34:13.141198332 +0000 UTC" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.544961 5800 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15037h45m51.596240856s" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.545795 5800 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.546842 5800 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.548459 5800 log.go:25] "Validated CRI v1 runtime API" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.552462 5800 log.go:25] "Validated CRI v1 image API" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.553532 5800 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 19:48:21.561685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.556146 5800 fs.go:135] Filesystem UUIDs: map[6a48c8a0-f570-4276-aa55-82c35a804ca1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ba7201a7-8181-4839-854e-e832b9e53533:/dev/nvme0n1p3] Nov 02 19:48:21.561940 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.556168 5800 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c8cbc0b3b6c6558a1e4256ba60bc23e37e9ad0f2f2af3538d52f899300c2d7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c8cbc0b3b6c6558a1e4256ba60bc23e37e9ad0f2f2af3538d52f899300c2d7b/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22ef7c50f8e0f13b678ac9707f06ae17ddc76632dc960700e715f04f44e58aac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22ef7c50f8e0f13b678ac9707f06ae17ddc76632dc960700e715f04f44e58aac/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d1b024470a51ec41f4cd244ffeb0f6d1665ed19789513a78e31ee72f2c08eb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d1b024470a51ec41f4cd244ffeb0f6d1665ed19789513a78e31ee72f2c08eb2/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31d628e29b74262ad921ff1bc39a244ca7786a157e0f80b051a5b1e79d403084/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31d628e29b74262ad921ff1bc39a244ca7786a157e0f80b051a5b1e79d403084/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c1913565262aaea69f7b1217d7f4e0bcb29f959991a42f7d4faafaa03fc2938/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c1913565262aaea69f7b1217d7f4e0bcb29f959991a42f7d4faafaa03fc2938/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/436ecc028f0783e87e73964081c3450fa555f6c621e23cfa51d56d65404575ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/436ecc028f0783e87e73964081c3450fa555f6c621e23cfa51d56d65404575ef/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4455d1c23aed6d355149b9f165eb7fd81392b806f926ca925d76203776d7886b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4455d1c23aed6d355149b9f165eb7fd81392b806f926ca925d76203776d7886b/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/483d11b57a7c5b92a6162154b952c246d5da36af585db874389d0091d1d96c61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/483d11b57a7c5b92a6162154b952c246d5da36af585db874389d0091d1d96c61/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/583ca4a14a9c3c9c0e2e22b0e585b398b5bbf9212e7c55820e2e58006c6b0351/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/583ca4a14a9c3c9c0e2e22b0e585b398b5bbf9212e7c55820e2e58006c6b0351/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58f1a311a9c9dc3d9144f6f549b585a3b17439db8b864c3ea821950fd89a4d5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58f1a311a9c9dc3d9144f6f549b585a3b17439db8b864c3ea821950fd89a4d5c/userdata/shm major:0 minor:441 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b309c160687ca5349bdb92e83e4a4e7f232131269379c381c3bfc54c1f89cc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b309c160687ca5349bdb92e83e4a4e7f232131269379c381c3bfc54c1f89cc0/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85b786d7f7f6161e291510ecae10ea6462c9dc23fe272a282ad908013cccfb57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85b786d7f7f6161e291510ecae10ea6462c9dc23fe272a282ad908013cccfb57/userdata/shm major:0 minor:445 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/881f780a8a58bd748681d528700d9fec13aee1c088482468c4a2b75aa5a23b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/881f780a8a58bd748681d528700d9fec13aee1c088482468c4a2b75aa5a23b85/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/886eee9e6338a661d858d979d3ef04f641715fa875aa1e1c99d7221445275b1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/886eee9e6338a661d858d979d3ef04f641715fa875aa1e1c99d7221445275b1a/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97153f35a89bd7557d75c9227d324d3dec78630194e4a8c6947fe2dcfc9c7cb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97153f35a89bd7557d75c9227d324d3dec78630194e4a8c6947fe2dcfc9c7cb8/userdata/shm major:0 minor:408 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad7ee0d0e6f7758a48042a1fafd594eb8d12952971209634878c3dc1817e7e45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad7ee0d0e6f7758a48042a1fafd594eb8d12952971209634878c3dc1817e7e45/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b994a73ee0369bbc0bb09f9675ed8f4ce063a5edea9cc12532d9e271df27a7fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b994a73ee0369bbc0bb09f9675ed8f4ce063a5edea9cc12532d9e271df27a7fb/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bca6f05f35d6071ff8afd2ce520c1438e201ac036ed294b5b08564f5dda2f145/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bca6f05f35d6071ff8afd2ce520c1438e201ac036ed294b5b08564f5dda2f145/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf0e2eb10c8cec711f8c1fba04e0fbc64030fe157ddf5f03312f81cee70aac4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf0e2eb10c8cec711f8c1fba04e0fbc64030fe157ddf5f03312f81cee70aac4a/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4c2b765b7a1efaa8ef7994476dec85ed89b4270b3433294f2412e65008050ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4c2b765b7a1efaa8ef7994476dec85ed89b4270b3433294f2412e65008050ab/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7889132659b1013de4f59658e7579684be8afe85340621477345acf2ec1d376/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7889132659b1013de4f59658e7579684be8afe85340621477345acf2ec1d376/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9b5284b4e60cf6be7d5f6e777159aee58564a7ca64537817b1c281a15ca0ec0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9b5284b4e60cf6be7d5f6e777159aee58564a7ca64537817b1c281a15ca0ec0/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e06b5369cade018e3a47d004cbdd99da6ecf45fa7e05f4a4323a93565d574863/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e06b5369cade018e3a47d004cbdd99da6ecf45fa7e05f4a4323a93565d574863/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4fac70cb6dc09857b70c7ff6bdf04a10c33ddd3311ada3153e3915f38c58041/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4fac70cb6dc09857b70c7ff6bdf04a10c33ddd3311ada3153e3915f38c58041/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea3430cdb385a0478c77b8bcc290714b265165e826ce14a7e16e729bd6641f54/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea3430cdb385a0478c77b8bcc290714b265165e826ce14a7e16e729bd6641f54/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4c0f792a4995f4c1b4cd4406f9b962ce6a321c5653108719d490513ef7a8a79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4c0f792a4995f4c1b4cd4406f9b962ce6a321c5653108719d490513ef7a8a79/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5d921a9e6dafbdcab26570eed2c4559f056b94e0338c43bf0c18ce4d5b319a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5d921a9e6dafbdcab26570eed2c4559f056b94e0338c43bf0c18ce4d5b319a5/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f805b4c40af57e21c99de8bf14aa5092df02445c92ca37b97e7d6cd3bf4c1220/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f805b4c40af57e21c99de8bf14aa5092df02445c92ca37b97e7d6cd3bf4c1220/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8fc59e78c7193eab0c62117404d42facf071020aef535cc2cdf95e1da9bf867/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8fc59e78c7193eab0c62117404d42facf071020aef535cc2cdf95e1da9bf867/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9f46b44620d281d218226a68c8fc2fbe6a90ea6665c3178f4c214555053193e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9f46b44620d281d218226a68c8fc2fbe6a90ea6665c3178f4c214555053193e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b/userdata/shm major:0 minor:443 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/kube-api-access-8h788:{mountpoint:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/kube-api-access-8h788 major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/registry-tls major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fb381a6-0d14-47a9-a144-4fd8be651947/volumes/kubernetes.io~projected/kube-api-access-wfsfs:{mountpoint:/var/lib/kubelet/pods/0fb381a6-0d14-47a9-a144-4fd8be651947/volumes/kubernetes.io~projected/kube-api-access-wfsfs major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fb381a6-0d14-47a9-a144-4fd8be651947/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0fb381a6-0d14-47a9-a144-4fd8be651947/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/184c44a1-6433-4fb3-a7cd-ad9fa39cb642/volumes/kubernetes.io~projected/kube-api-access-zxznp:{mountpoint:/var/lib/kubelet/pods/184c44a1-6433-4fb3-a7cd-ad9fa39cb642/volumes/kubernetes.io~projected/kube-api-access-zxznp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a620cde-dc26-4159-8e40-41fdd982d61f/volumes/kubernetes.io~projected/kube-api-access-ffnqz:{mountpoint:/var/lib/kubelet/pods/1a620cde-dc26-4159-8e40-41fdd982d61f/volumes/kubernetes.io~projected/kube-api-access-ffnqz major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a620cde-dc26-4159-8e40-41fdd982d61f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1a620cde-dc26-4159-8e40-41fdd982d61f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/207974a1-670c-49f2-b0fb-3946d261e608/volumes/kubernetes.io~projected/kube-api-access-jkwfw:{mountpoint:/var/lib/kubelet/pods/207974a1-670c-49f2-b0fb-3946d261e608/volumes/kubernetes.io~projected/kube-api-access-jkwfw major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31464cb4-534f-41a2-bc06-99343f47a433/volumes/kubernetes.io~projected/kube-api-access-tm8lh:{mountpoint:/var/lib/kubelet/pods/31464cb4-534f-41a2-bc06-99343f47a433/volumes/kubernetes.io~projected/kube-api-access-tm8lh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b79f192-7b15-47d5-91b4-1772b3541ecf/volumes/kubernetes.io~projected/kube-api-access-6cxmg:{mountpoint:/var/lib/kubelet/pods/3b79f192-7b15-47d5-91b4-1772b3541ecf/volumes/kubernetes.io~projected/kube-api-access-6cxmg major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b79f192-7b15-47d5-91b4-1772b3541ecf/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3b79f192-7b15-47d5-91b4-1772b3541ecf/volumes/kubernetes.io~secret/cert major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426ee25e-f4e1-44da-b11f-191d799f8f27/volumes/kubernetes.io~projected/kube-api-access-zjt2l:{mountpoint:/var/lib/kubelet/pods/426ee25e-f4e1-44da-b11f-191d799f8f27/volumes/kubernetes.io~projected/kube-api-access-zjt2l major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b70b3c0-c849-4af0-904f-677aa9222a1e/volumes/kubernetes.io~projected/kube-api-access-2xdkl:{mountpoint:/var/lib/kubelet/pods/4b70b3c0-c849-4af0-904f-677aa9222a1e/volumes/kubernetes.io~projected/kube-api-access-2xdkl major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/505ade63-4db5-4709-9e6a-cdaed488b662/volumes/kubernetes.io~projected/kube-api-access-gc2hd:{mountpoint:/var/lib/kubelet/pods/505ade63-4db5-4709-9e6a-cdaed488b662/volumes/kubernetes.io~projected/kube-api-access-gc2hd major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e413f92-3561-4f46-a45b-1fbe4919af71/volumes/kubernetes.io~projected/kube-api-access-hqs4h:{mountpoint:/var/lib/kubelet/pods/5e413f92-3561-4f46-a45b-1fbe4919af71/volumes/kubernetes.io~projected/kube-api-access-hqs4h major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/674789e3-135f-4296-9de6-274e37588b04/volumes/kubernetes.io~projected/kube-api-access-5bh5c:{mountpoint:/var/lib/kubelet/pods/674789e3-135f-4296-9de6-274e37588b04/volumes/kubernetes.io~projected/kube-api-access-5bh5c major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~projected/kube-api-access-r47cn:{mountpoint:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~projected/kube-api-access-r47cn major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/default-certificate major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/stats-auth major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/703df830-74cd-46d0-96ab-ab6b2b42c439/volumes/kubernetes.io~projected/kube-api-access-jnjmr:{mountpoint:/var/lib/kubelet/pods/703df830-74cd-46d0-96ab-ab6b2b42c439/volumes/kubernetes.io~projected/kube-api-access-jnjmr major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/703df830-74cd-46d0-96ab-ab6b2b42c439/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/703df830-74cd-46d0-96ab-ab6b2b42c439/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volumes/kubernetes.io~projected/kube-api-access-2z9vr:{mountpoint:/var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volumes/kubernetes.io~projected/kube-api-access-2z9vr major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bd90b6c-66f9-4ea8-8f73-2981acce43db/volumes/kubernetes.io~projected/kube-api-access-rnnnd:{mountpoint:/var/lib/kubelet/pods/7bd90b6c-66f9-4ea8-8f73-2981acce43db/volumes/kubernetes.io~projected/kube-api-access-rnnnd major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8faa206c-23f9-4f03-8b85-d66e47f08c30/volumes/kubernetes.io~projected/kube-api-access-lm8b7:{mountpoint:/var/lib/kubelet/pods/8faa206c-23f9-4f03-8b85-d66e47f08c30/volumes/kubernetes.io~projected/kube-api-access-lm8b7 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9271574f-8b74-4de7-8bcf-f90951a5fd4a/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/9271574f-8b74-4de7-8bcf-f90951a5fd4a/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97562254-d922-4415-9438-ae340b218942/volumes/kubernetes.io~projected/kube-api-access-tn7xb:{mountpoint:/var/lib/kubelet/pods/97562254-d922-4415-9438-ae340b218942/volumes/kubernetes.io~projected/kube-api-access-tn7xb major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97562254-d922-4415-9438-ae340b218942/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/97562254-d922-4415-9438-ae340b218942/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a36272f8-3fbc-4501-b03c-91a343e37204/volumes/kubernetes.io~projected/kube-api-access-j6b95:{mountpoint:/var/lib/kubelet/pods/a36272f8-3fbc-4501-b03c-91a343e37204/volumes/kubernetes.io~projected/kube-api-access-j6b95 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f/volumes/kubernetes.io~projected/kube-api-access-jhtw5:{mountpoint:/var/lib/kubelet/pods/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f/volumes/kubernetes.io~projected/kube-api-access-jhtw5 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21/volumes/kubernetes.io~projected/kube-api-access-6xs6q:{mountpoint:/var/lib/kubelet/pods/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21/volumes/kubernetes.io~projected/kube-api-access-6xs6q major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~projected/kube-api-access-qpj8m:{mountpoint:/var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~projected/kube-api-access-qpj8m major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb5fc542-006a-4da5-b985-cc7b0f1709a7/volumes/kubernetes.io~projected/kube-api-access-8mcnh:{mountpoint:/var/lib/kubelet/pods/cb5fc542-006a-4da5-b985-cc7b0f1709a7/volumes/kubernetes.io~projected/kube-api-access-8mcnh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbaf72f9-c2ab-4081-87f4-7747ffc180ad/volumes/kubernetes.io~projected/kube-api-access-r6txz:{mountpoint:/var/lib/kubelet/pods/dbaf72f9-c2ab-4081-87f4-7747ffc180ad/volumes/kubernetes.io~projected/kube-api-access-r6txz major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbaf72f9-c2ab-4081-87f4-7747ffc180ad/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/dbaf72f9-c2ab-4081-87f4-7747ffc180ad/volumes/kubernetes.io~secret/signing-key major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deaf3243-1eeb-4d94-bef9-524b67035ba2/volumes/kubernetes.io~projected/kube-api-access-sdr6s:{mountpoint:/var/lib/kubelet/pods/deaf3243-1eeb-4d94-bef9-524b67035ba2/volumes/kubernetes.io~projected/kube-api-access-sdr6s major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deaf3243-1eeb-4d94-bef9-524b67035ba2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/deaf3243-1eeb-4d94-bef9-524b67035ba2/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5640e81-834c-4854-9cc9-589ccbd39d64/volumes/kubernetes.io~projected/kube-api-access-s9m9j:{mountpoint:/var/lib/kubelet/pods/e5640e81-834c-4854-9cc9-589ccbd39d64/volumes/kubernetes.io~projected/kube-api-access-s9m9j major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eceae17b-5b0b-45ce-87cf-9d720583c247/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/eceae17b-5b0b-45ce-87cf-9d720583c247/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f40f9013-d584-4546-940b-82f31a20cd4f/volumes/kubernetes.io~projected/kube-api-access-55k45:{mountpoint:/var/lib/kubelet/pods/f40f9013-d584-4546-940b-82f31a20cd4f/volumes/kubernetes.io~projected/kube-api-access-55k45 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f98defbd-aa11-4a76-bac3-4d7aacdb44a1/volumes/kubernetes.io~projected/kube-api-access-7ggbz:{mountpoint:/var/lib/kubelet/pods/f98defbd-aa11-4a76-bac3-4d7aacdb44a1/volumes/kubernetes.io~projected/kube-api-access-7ggbz major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/kube-api-access-jckjq:{mountpoint:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/kube-api-access-jckjq major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/registry-tls major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fea63025-526f-4fd8-b303-bc115c1226dd/volumes/kubernetes.io~projected/kube-api-access-mf4qd:{mountpoint:/var/lib/kubelet/pods/fea63025-526f-4fd8-b303-bc115c1226dd/volumes/kubernetes.io~projected/kube-api-access-mf4qd major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fea63025-526f-4fd8-b303-bc115c1226dd/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/fea63025-526f-4fd8-b303-bc115c1226dd/volumes/kubernetes.io~secret/metrics-certs major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff626920-a094-4782-aae1-ac6bc9d0b359/volumes/kubernetes.io~projected/kube-api-access-ct6x2:{mountpoint:/var/lib/kubelet/pods/ff626920-a094-4782-aae1-ac6bc9d0b359/volumes/kubernetes.io~projected/kube-api-access-ct6x2 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff626920-a094-4782-aae1-ac6bc9d0b359/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ff626920-a094-4782-aae1-ac6bc9d0b359/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f9453d2585988c55e3d1373229abe85e52573313d14c80bdbed15d8764784a3c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c54bad97cf0fce2bce839d899efaae7cb4ce44fc68df7596789df1f0f2b20a0f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/eeca5d6d076bf52e4497a3460c35e634a31ee5c5cbcd409dccaa94eb11b742bb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9b0035839a32679f2ce1fb41ba45d8aa8bef9dacc930ebe45fc272520b0fcf3e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7832640067f657d2e6e0dcb126bb566f39104db8d0a12ed79a939eec9e997444/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/18aa8743bf4aed2e11519e2e1fb8d5ba5eb0949d934907f31b8dd4fe0017e219/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8886675d706c941fe2c94ecbf77cb6740de0cd3ec9fa6315f400cf0321c116e3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d4d88b70b7f48891a749e0b63e18ea7cc67b4d932f69dbac15dc1005931e3923/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/656b59ae7fad8dbfabc4d6c8996f9240f1621b726e13e3b6ef532cc5a15c2fd3/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/9a0e9620ab568020c5066f606359d031a465db283aacf9d74f28ebf13ddce813/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/6a68706202f96e3661353397372c49fb993de611790b62ad7f86c57d4282c049/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c0064dbfce4280c1c94e5b733629e474349fa7189d65ad39eafa85df1dda2ee2/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/156979e610eec790a007b3816dbcdd04f6cdd5e7b16a576955e7410bd968a421/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ab211a8921dd0c66d48b4737a0d776bff6e89b3f59d9bbee9a0bffe0b589d391/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/daa2740322fc95dd2e0201c3decbdf3242e8f0e54697205d13c6506c4ea538e9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/57d494e08f7a421f343d5415dc9433c50d8608b9cbbf95fcc4564e816562371c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/322af582d57181072eb6faa9d9768d679ae548f7c477c7111aa29a449705f850/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5527d361a7928ec7e4275ecdca50c906b8d9449c8825a9a9017b08bbbd543c9f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/72448b9f5789cefbf8e3a5dffb92f02bce26e14f01b910bf7f97e63ab1a02863/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/49259904ed65124c1eec210e437eed6f81246dfcec34ca87b7cdca490530aadc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/04d404ed0dd13779067626c3d9d582085fb70c37eb92a9a99a04491a074a7532/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/f0457ca98779eab7bd2333af7f36f0b55c96998e7bd32d3d6e08b34ef311c220/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c318b6f43542c73f4aafbac235193c030cfdd73b70bfba07375d5ad778486bd7/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/50cc14681df600485d6e42fadf8351a60f4f3c7f01745e208ce589919e905145/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/005d5b078ad7414ca05879dc03d5a2bbd95fb73448848f17b9fdcb7dcae3dc24/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/dc2a02998a47b37b18ae08fb2ee38e45676eeb1146531f2a4b925b6b2877f7a2/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/6a59e3353f409091fd5afb35f294914e03df0312466e7380f7f4e4eb54993d14/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/ff934dccd73f00b5eddd80cff5b6b8c5e7485b0afb03405e15ea3c54dfc1896e/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/0f5725f271a6edba433825210c65a571399a15bcf9f5780ff55606708683c811/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/cf0fbeccc8cd60535b759c0086653b3395a9aefa82933c6b38f664eb1c68eb00/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/37d1566861b2f8a9eecd5a8888139a1ac5c69a350848748444df8451b478a032/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/4020e781887750871ad6a0788ebd54351f80e9f73522b4ef1d809fdae65072c4/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/b7170726c5eae6a4c850615ffb01ded3f8a72139de399fe2d0fd6d707f6542cc/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/905d221e0592d12cca566a52e1e975e8c7497fe8521fc1a18b15db3998751eb2/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/5c97eada364f1792ea53877c62748c3b6a56c94c2718006c9e48ea41b9b121a8/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/d2c0150cd752387c56bdc2592b098739133d9b48890da444855254c5399495f6/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/2e7734832794afa597d1178814d1fce8ae3bf48bfab27f49d2c1a0865b63ee62/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a2af05834e598f2171a8146fac825b203f3cde360dd27f44d1110778bd7011e1/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/2af2fa5db217a9f6c85499ac4b42cab14caffb9e9a772256cddbf76db1ab31cb/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/50321a88e3daa840534dcaa3525b0a8d0b37a40cfcebc791f45ea19e79d09f0f/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/58066adf335c7fdad2b617d2be338ad727ba7c4ca91b49ef59d6280ffc9c279b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/62e33f12efbdd5db6dd8e142ebb5f9d063b68224cdffe1c1a70204f30facc681/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/42bb2524eaa667ea48359bf0c1ec7ee4f7c7d0949f697b0c54884aa1141c4922/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/6af0dcf6800d1dc83093e09159869d1ab2fc13849ebb9f3f21537c7459ef5329/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/a77dae472e7a9d96cfd6894f92b8819cf1881edcb3e858173a6498af002763de/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/ea4beee29a60ab5fd0fcdab0d3adf7e26c32b19cd24dcdb1a2411352238089f9/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a9ca5ded4b8b68a2e01eb01c9448799a35ba0684da0d0d73cffccca2c555676a/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/196ade29ecb550a69f09c38a9e9575c42db2c3fe14778c658aa32f2168cc29b3/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/474b43ec500bb9f307ea6dc7386d28576fe91675ceb7ea57be42c2e19de2b7a9/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/604e38303d0b7b1c014690f6ed6411aaec54ec757d4ee4ee9e4083c948292b65/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/e597db139618edcaf1a6c4e5dbcb05e489a5a8e0b02882b6a21833f11f2543ad/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/418ee27e9700eb48e83ac20d7f4f89ab6b422cb9afefaad03cdb7b5ec9edd3f1/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/0f571b5e046e60f949510f57c2d78fa0908e449fbf7608fc97a1196d5a5c553a/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/9b3aaffea49ba591d06445c64680dcedd6350fd1298d0d05adaf8aac6c67cf80/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/4e272949315129982144a1ae44d122aa853a387e58d74823aa11b2670736d256/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/9d83087bf0c90e3b4c0dcc02208993c4c467176f665078d553ae7f18e8aad7d9/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/7a1fe848c7d839d1ad2f4d684aac4f5731bf0a750cb2f5fcc6c36110b29029fb/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/7b8a7b9df6c0e15c729d67bcd3370e4b693a953803a6af6da27278dc8cffd7ad/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/cb57e7e06cf374e8f803e52d50972f29023b27814299008ec2b70f2b6aa9bff1/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/c5d98be527823791d5b5346fa02805ad326f5c500d8f69b491be8801e9a336e5/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/5a792cd18529bd18dd31ada754cfe3383241c641379036f27560eea43e52fecd/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/d8db5012fb01db580ae05e5240797aa8a20a0b5ef3ca4ecd7631058b361dc3db/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/dd1e88576a7fda847b62ef1dbb80130ff0328a923a759e27a9ae97ce2d59c104/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/19ea9c45b7ae30c0ff6743c5024df27a1f268caad265de52ae5ac809da7ef6a0/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/c4b17499212f830baa470bc16bff93fff06289f68ee5f97dcfb0e4e8b883f776/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/4a93b1ece7db8bf7af69a43122cd9238d53a8798becee93836f5ba409d223d08/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/aa7bf28daf514de3833b5dd2346c9f8dc1113476b98a41bf8fc8fea9be4a3a42/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/011d0aea5b89c4721629d1883b2d566e5b6836355f30912ee671b82dda7e07bf/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/4f4f8191ef67bfef73651619647712ca113be205f5d64505e0c5f02c6576d37d/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/a5cd4b150b5b5cd8b0423674cd3aac1b1fd95f62d661106f33b684b444cafd88/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/cc9333aa19a4414e6952b7bd446036cc2ee0a3ee80a6f55a55c4a4a8f53ccf34/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/499c7794d8a9778cd79bd6fe5bd73fa10c422941ef30bd658b8c74907a7718de/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/64e4f37dde13fc3e5841a8640138923f3c1c439a34a4c5b7b44a3577bacecebf/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c5faae4b755a8d98878f368989f2280c1486a33dc63e1a3c53eedac519b4f15e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6be8306ff279467254613434446b5b06e89654afdb77e2a9c53ce87cd4960ba0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bc1ee49a85aaae92845bd5a505a01c7456a136c45af41952e803a9949b0d1a1b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6539dd63e6a2c26146ff710ee710cbf6d541dd005f8a60cc5bf624940c0c2daf/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/229d3f69ccf02556aa9deb8ffe81d18d58f572e1f71067b86f944142da2d8c6e/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/76a6ee9247b989fd3b19c742cb9f3354257f80925bf5868535ca6b4a6cbed170/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f91e0781dae2d6ee49580866966b745d168e39273f1d0b39e3f2847c2288d5f2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6449439e5c20a88943d6e90faf91d8c8d75f1bb4608cc4626b923be8ef615ee3/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 19:48:21.566926 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.566217 5800 manager.go:217] Machine: {Timestamp:2026-11-02 19:48:21.56512745 +0000 UTC m=+0.081900026 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084709 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25d5e14844b44bedf0adbba0e79914 SystemUUID:ec25d5e1-4844-b44b-edf0-adbba0e79914 BootID:7eedd5d0-d9ea-4fb9-a3fe-4d7a9e0f8e58 Filesystems:[{Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8fc59e78c7193eab0c62117404d42facf071020aef535cc2cdf95e1da9bf867/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b/userdata/shm DeviceMajor:0 DeviceMinor:443 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/207974a1-670c-49f2-b0fb-3946d261e608/volumes/kubernetes.io~projected/kube-api-access-jkwfw DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf0e2eb10c8cec711f8c1fba04e0fbc64030fe157ddf5f03312f81cee70aac4a/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22ef7c50f8e0f13b678ac9707f06ae17ddc76632dc960700e715f04f44e58aac/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbaf72f9-c2ab-4081-87f4-7747ffc180ad/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbaf72f9-c2ab-4081-87f4-7747ffc180ad/volumes/kubernetes.io~projected/kube-api-access-r6txz DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b309c160687ca5349bdb92e83e4a4e7f232131269379c381c3bfc54c1f89cc0/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/703df830-74cd-46d0-96ab-ab6b2b42c439/volumes/kubernetes.io~projected/kube-api-access-jnjmr DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/kube-api-access-8h788 DeviceMajor:0 DeviceMinor:406 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31464cb4-534f-41a2-bc06-99343f47a433/volumes/kubernetes.io~projected/kube-api-access-tm8lh DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f98defbd-aa11-4a76-bac3-4d7aacdb44a1/volumes/kubernetes.io~projected/kube-api-access-7ggbz DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/886eee9e6338a661d858d979d3ef04f641715fa875aa1e1c99d7221445275b1a/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21/volumes/kubernetes.io~projected/kube-api-access-6xs6q DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad7ee0d0e6f7758a48042a1fafd594eb8d12952971209634878c3dc1817e7e45/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b70b3c0-c849-4af0-904f-677aa9222a1e/volumes/kubernetes.io~projected/kube-api-access-2xdkl DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:396 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eceae17b-5b0b-45ce-87cf-9d720583c247/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9f46b44620d281d218226a68c8fc2fbe6a90ea6665c3178f4c214555053193e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/483d11b57a7c5b92a6162154b952c246d5da36af585db874389d0091d1d96c61/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deaf3243-1eeb-4d94-bef9-524b67035ba2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff626920-a094-4782-aae1-ac6bc9d0b359/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a620cde-dc26-4159-8e40-41fdd982d61f/volumes/kubernetes.io~projected/kube-api-access-ffnqz DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7bd90b6c-66f9-4ea8-8f73-2981acce43db/volumes/kubernetes.io~projected/kube-api-access-rnnnd DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c1913565262aaea69f7b1217d7f4e0bcb29f959991a42f7d4faafaa03fc2938/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97153f35a89bd7557d75c9227d324d3dec78630194e4a8c6947fe2dcfc9c7cb8/userdata/shm DeviceMajor:0 DeviceMinor:408 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volumes/kubernetes.io~projected/kube-api-access-2z9vr DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5640e81-834c-4854-9cc9-589ccbd39d64/volumes/kubernetes.io~projected/kube-api-access-s9m9j DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fb381a6-0d14-47a9-a144-4fd8be651947/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:401 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fea63025-526f-4fd8-b303-bc115c1226dd/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97562254-d922-4415-9438-ae340b218942/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~projected/kube-api-access-qpj8m DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d1b024470a51ec41f4cd244ffeb0f6d1665ed19789513a78e31ee72f2c08eb2/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/97562254-d922-4415-9438-ae340b218942/volumes/kubernetes.io~projected/kube-api-access-tn7xb DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f/volumes/kubernetes.io~projected/kube-api-access-jhtw5 DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58f1a311a9c9dc3d9144f6f549b585a3b17439db8b864c3ea821950fd89a4d5c/userdata/shm DeviceMajor:0 DeviceMinor:441 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:405 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fea63025-526f-4fd8-b303-bc115c1226dd/volumes/kubernetes.io~projected/kube-api-access-mf4qd DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bca6f05f35d6071ff8afd2ce520c1438e201ac036ed294b5b08564f5dda2f145/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~projected/kube-api-access-r47cn DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f805b4c40af57e21c99de8bf14aa5092df02445c92ca37b97e7d6cd3bf4c1220/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5d921a9e6dafbdcab26570eed2c4559f056b94e0338c43bf0c18ce4d5b319a5/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7889132659b1013de4f59658e7579684be8afe85340621477345acf2ec1d376/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a63760e-f41b-4958-83a5-e4944090dccb/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:407 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb5fc542-006a-4da5-b985-cc7b0f1709a7/volumes/kubernetes.io~projected/kube-api-access-8mcnh DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fb381a6-0d14-47a9-a144-4fd8be651947/volumes/kubernetes.io~projected/kube-api-access-wfsfs DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff626920-a094-4782-aae1-ac6bc9d0b359/volumes/kubernetes.io~projected/kube-api-access-ct6x2 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4c0f792a4995f4c1b4cd4406f9b962ce6a321c5653108719d490513ef7a8a79/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e413f92-3561-4f46-a45b-1fbe4919af71/volumes/kubernetes.io~projected/kube-api-access-hqs4h DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/kube-api-access-jckjq DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/583ca4a14a9c3c9c0e2e22b0e585b398b5bbf9212e7c55820e2e58006c6b0351/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4455d1c23aed6d355149b9f165eb7fd81392b806f926ca925d76203776d7886b/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/703df830-74cd-46d0-96ab-ab6b2b42c439/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deaf3243-1eeb-4d94-bef9-524b67035ba2/volumes/kubernetes.io~projected/kube-api-access-sdr6s DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4c2b765b7a1efaa8ef7994476dec85ed89b4270b3433294f2412e65008050ab/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea3430cdb385a0478c77b8bcc290714b265165e826ce14a7e16e729bd6641f54/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c8cbc0b3b6c6558a1e4256ba60bc23e37e9ad0f2f2af3538d52f899300c2d7b/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a620cde-dc26-4159-8e40-41fdd982d61f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31d628e29b74262ad921ff1bc39a244ca7786a157e0f80b051a5b1e79d403084/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b994a73ee0369bbc0bb09f9675ed8f4ce063a5edea9cc12532d9e271df27a7fb/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a36272f8-3fbc-4501-b03c-91a343e37204/volumes/kubernetes.io~projected/kube-api-access-j6b95 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426ee25e-f4e1-44da-b11f-191d799f8f27/volumes/kubernetes.io~projected/kube-api-access-zjt2l DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/caf2ae77-8244-44fa-a47f-1b8555fed1f2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f40f9013-d584-4546-940b-82f31a20cd4f/volumes/kubernetes.io~projected/kube-api-access-55k45 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/881f780a8a58bd748681d528700d9fec13aee1c088482468c4a2b75aa5a23b85/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9271574f-8b74-4de7-8bcf-f90951a5fd4a/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:438 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b79f192-7b15-47d5-91b4-1772b3541ecf/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/184c44a1-6433-4fb3-a7cd-ad9fa39cb642/volumes/kubernetes.io~projected/kube-api-access-zxznp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4fac70cb6dc09857b70c7ff6bdf04a10c33ddd3311ada3153e3915f38c58041/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/436ecc028f0783e87e73964081c3450fa555f6c621e23cfa51d56d65404575ef/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8faa206c-23f9-4f03-8b85-d66e47f08c30/volumes/kubernetes.io~projected/kube-api-access-lm8b7 DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b79f192-7b15-47d5-91b4-1772b3541ecf/volumes/kubernetes.io~projected/kube-api-access-6cxmg DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/505ade63-4db5-4709-9e6a-cdaed488b662/volumes/kubernetes.io~projected/kube-api-access-gc2hd DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85b786d7f7f6161e291510ecae10ea6462c9dc23fe272a282ad908013cccfb57/userdata/shm DeviceMajor:0 DeviceMinor:445 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9b5284b4e60cf6be7d5f6e777159aee58564a7ca64537817b1c281a15ca0ec0/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e06b5369cade018e3a47d004cbdd99da6ecf45fa7e05f4a4323a93565d574863/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/674789e3-135f-4296-9de6-274e37588b04/volumes/kubernetes.io~projected/kube-api-access-5bh5c DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c8cbc0b3b6c655 MacAddress:36:19:ef:f9:1a:fa Speed:10000 Mtu:8901} {Name:22ef7c50f8e0f13 MacAddress:ce:6a:85:68:c6:f4 Speed:10000 Mtu:8901} {Name:28e75b5fd0566e2 MacAddress:92:b1:4b:ee:fd:05 Speed:10000 Mtu:8901} {Name:31d628e29b74262 MacAddress:8e:6d:a3:85:04:f1 Speed:10000 Mtu:8901} {Name:3c1913565262aae MacAddress:4e:49:e5:eb:72:ec Speed:10000 Mtu:8901} {Name:436ecc028f0783e MacAddress:ae:22:68:3d:e0:c8 Speed:10000 Mtu:8901} {Name:4455d1c23aed6d3 MacAddress:36:fd:23:4f:0d:f8 Speed:10000 Mtu:8901} {Name:483d11b57a7c5b9 MacAddress:ca:3a:63:c8:1c:a7 Speed:10000 Mtu:8901} {Name:583ca4a14a9c3c9 MacAddress:06:92:4a:28:df:6a Speed:10000 Mtu:8901} {Name:58f1a311a9c9dc3 MacAddress:de:d5:43:50:8e:b6 Speed:10000 Mtu:8901} {Name:5b309c160687ca5 MacAddress:4e:ad:60:0e:b0:29 Speed:10000 Mtu:8901} {Name:85b786d7f7f6161 MacAddress:c6:15:d3:07:18:f7 Speed:10000 Mtu:8901} {Name:881f780a8a58bd7 MacAddress:62:f8:77:1b:fa:33 Speed:10000 Mtu:8901} {Name:886eee9e6338a66 MacAddress:5e:c0:0a:2c:e3:1c Speed:10000 Mtu:8901} {Name:97153f35a89bd75 MacAddress:7a:4d:49:49:7e:f9 Speed:10000 Mtu:8901} {Name:ad7ee0d0e6f7758 MacAddress:76:e1:62:1d:f9:99 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:da:fc:81:b3 Speed:0 Mtu:9001} {Name:br-int MacAddress:52:aa:4d:32:f1:93 Speed:0 Mtu:8901} {Name:cf0e2eb10c8cec7 MacAddress:ea:8f:27:b2:2d:d0 Speed:10000 Mtu:8901} {Name:d4c2b765b7a1efa MacAddress:ee:88:c4:fa:c7:a2 Speed:10000 Mtu:8901} {Name:d7889132659b101 MacAddress:12:87:9a:ef:b9:76 Speed:10000 Mtu:8901} {Name:ea3430cdb385a04 MacAddress:1a:10:d5:71:29:b7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:da:fc:81:b3 Speed:0 Mtu:9001} {Name:f5d921a9e6dafbd MacAddress:b2:7f:1f:b5:95:00 Speed:10000 Mtu:8901} {Name:f805b4c40af57e2 MacAddress:42:c9:1f:ba:bc:ba Speed:10000 Mtu:8901} {Name:f8fc59e78c7193e MacAddress:6a:e0:eb:0e:61:56 Speed:10000 Mtu:8901} {Name:fb3c12577c50a90 MacAddress:0e:34:55:62:21:31 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:be:8c:52:c2:e4:ee Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:16:d9:91:4c:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 19:48:21.566926 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.566932 5800 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 19:48:21.567257 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.566999 5800 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 19:48:21.567417 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567322 5800 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 19:48:21.567491 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567352 5800 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-13.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 19:48:21.567538 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567494 5800 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 19:48:21.567538 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567502 5800 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 19:48:21.567538 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567513 5800 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:48:21.567538 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567532 5800 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 19:48:21.567640 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567627 5800 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 19:48:21.567801 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567791 5800 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 19:48:21.567916 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567907 5800 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 19:48:21.567956 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567946 5800 kubelet.go:493] "Attempting to sync node with API server" Nov 02 19:48:21.567984 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567964 5800 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 19:48:21.567984 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567975 5800 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 19:48:21.567984 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567984 5800 kubelet.go:405] "Adding apiserver pod source" Nov 02 19:48:21.568061 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.567992 5800 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 19:48:21.568948 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.568932 5800 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:48:21.568987 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.568950 5800 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:48:21.569507 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.569482 5800 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 19:48:21.572255 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572240 5800 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 19:48:21.572255 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572255 5800 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 19:48:21.572377 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572366 5800 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 19:48:21.572563 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572552 5800 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 19:48:21.572594 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572580 5800 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 19:48:21.572674 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572664 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 19:48:21.572710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572679 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 19:48:21.572710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572686 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 19:48:21.572710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572692 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 19:48:21.572710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572698 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 19:48:21.572710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572703 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 19:48:21.572710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572709 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 19:48:21.572904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572715 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 19:48:21.572904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572722 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 19:48:21.572904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572727 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 19:48:21.572904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572742 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 19:48:21.572904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572752 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 19:48:21.572904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572774 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 19:48:21.572904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.572780 5800 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 19:48:21.576161 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.576146 5800 server.go:1263] "Started kubelet" Nov 02 19:48:21.576586 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.576244 5800 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 19:48:21.576586 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.576388 5800 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 19:48:21.576815 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.576668 5800 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 19:48:21.576822 ip-10-0-134-13 systemd[1]: Started Kubernetes Kubelet. Nov 02 19:48:21.577005 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.576762 5800 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 19:48:21.580844 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.579846 5800 server.go:317] "Adding debug handlers to kubelet server" Nov 02 19:48:21.583304 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.583268 5800 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 19:48:21.584329 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.583318 5800 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 19:48:21.584329 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.583411 5800 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 19:48:21.584329 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.583424 5800 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 19:48:21.584329 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.583423 5800 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:48:21.584329 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.583538 5800 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 19:48:21.584329 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.583385 5800 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 19:42:09 +0000 UTC" deadline="2028-04-05 14:00:10.450858519 +0000 UTC" Nov 02 19:48:21.584329 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.583560 5800 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12474h11m48.867301764s" Nov 02 19:48:21.585559 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.585533 5800 factory.go:55] Registering systemd factory Nov 02 19:48:21.585644 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.585568 5800 factory.go:223] Registration of the systemd container factory successfully Nov 02 19:48:21.586249 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.586235 5800 factory.go:153] Registering CRI-O factory Nov 02 19:48:21.586320 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.586251 5800 factory.go:223] Registration of the crio container factory successfully Nov 02 19:48:21.586387 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.586326 5800 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 19:48:21.586387 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.586354 5800 factory.go:103] Registering Raw factory Nov 02 19:48:21.586387 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.586383 5800 manager.go:1199] Started watching for new ooms in manager Nov 02 19:48:21.586774 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.586760 5800 manager.go:319] Starting recovery of all containers Nov 02 19:48:21.587663 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587622 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="207974a1-670c-49f2-b0fb-3946d261e608" volumeName="kubernetes.io/projected/207974a1-670c-49f2-b0fb-3946d261e608-kube-api-access-jkwfw" seLinuxMountContext="" Nov 02 19:48:21.587746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587671 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b70b3c0-c849-4af0-904f-677aa9222a1e" volumeName="kubernetes.io/projected/4b70b3c0-c849-4af0-904f-677aa9222a1e-kube-api-access-2xdkl" seLinuxMountContext="" Nov 02 19:48:21.587746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587689 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674789e3-135f-4296-9de6-274e37588b04" volumeName="kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle" seLinuxMountContext="" Nov 02 19:48:21.587746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587704 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77546eb8-8ef9-46b1-a47a-ca80ca5f76ee" volumeName="kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-env-overrides" seLinuxMountContext="" Nov 02 19:48:21.587746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587717 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caf2ae77-8244-44fa-a47f-1b8555fed1f2" volumeName="kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-tmp" seLinuxMountContext="" Nov 02 19:48:21.587746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587729 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration" seLinuxMountContext="" Nov 02 19:48:21.587746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587744 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b79f192-7b15-47d5-91b4-1772b3541ecf" volumeName="kubernetes.io/projected/3b79f192-7b15-47d5-91b4-1772b3541ecf-kube-api-access-6cxmg" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587758 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674789e3-135f-4296-9de6-274e37588b04" volumeName="kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587780 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f73e2f3-6f98-41bd-abec-b72d60ac2bb5" volumeName="kubernetes.io/projected/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-kube-api-access-r47cn" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587795 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f" volumeName="kubernetes.io/secret/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-serving-cert" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587825 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21" volumeName="kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-daemon-config" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587841 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb5fc542-006a-4da5-b985-cc7b0f1709a7" volumeName="kubernetes.io/projected/cb5fc542-006a-4da5-b985-cc7b0f1709a7-kube-api-access-8mcnh" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587859 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-installation-pull-secrets" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587873 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77546eb8-8ef9-46b1-a47a-ca80ca5f76ee" volumeName="kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-script-lib" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587892 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbaf72f9-c2ab-4081-87f4-7747ffc180ad" volumeName="kubernetes.io/projected/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-kube-api-access-r6txz" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587905 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eceae17b-5b0b-45ce-87cf-9d720583c247" volumeName="kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587935 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587967 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fea63025-526f-4fd8-b303-bc115c1226dd" volumeName="kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587982 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-bound-sa-token" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.587995 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703df830-74cd-46d0-96ab-ab6b2b42c439" volumeName="kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588009 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97562254-d922-4415-9438-ae340b218942" volumeName="kubernetes.io/configmap/97562254-d922-4415-9438-ae340b218942-telemetry-config" seLinuxMountContext="" Nov 02 19:48:21.588028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588023 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff626920-a094-4782-aae1-ac6bc9d0b359" volumeName="kubernetes.io/configmap/ff626920-a094-4782-aae1-ac6bc9d0b359-config" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588038 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff626920-a094-4782-aae1-ac6bc9d0b359" volumeName="kubernetes.io/projected/ff626920-a094-4782-aae1-ac6bc9d0b359-kube-api-access-ct6x2" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588056 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/empty-dir/0a63760e-f41b-4958-83a5-e4944090dccb-ca-trust-extracted" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588073 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb381a6-0d14-47a9-a144-4fd8be651947" volumeName="kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-trusted-ca-bundle" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588091 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a620cde-dc26-4159-8e40-41fdd982d61f" volumeName="kubernetes.io/empty-dir/1a620cde-dc26-4159-8e40-41fdd982d61f-tmp-dir" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588113 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a620cde-dc26-4159-8e40-41fdd982d61f" volumeName="kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588126 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="207974a1-670c-49f2-b0fb-3946d261e608" volumeName="kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-binary-copy" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588155 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e413f92-3561-4f46-a45b-1fbe4919af71" volumeName="kubernetes.io/projected/5e413f92-3561-4f46-a45b-1fbe4919af71-kube-api-access-hqs4h" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588168 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f73e2f3-6f98-41bd-abec-b72d60ac2bb5" volumeName="kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-default-certificate" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588184 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bd90b6c-66f9-4ea8-8f73-2981acce43db" volumeName="kubernetes.io/projected/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-api-access-rnnnd" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588199 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb381a6-0d14-47a9-a144-4fd8be651947" volumeName="kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-tmp" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588213 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb381a6-0d14-47a9-a144-4fd8be651947" volumeName="kubernetes.io/projected/0fb381a6-0d14-47a9-a144-4fd8be651947-kube-api-access-wfsfs" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588230 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426ee25e-f4e1-44da-b11f-191d799f8f27" volumeName="kubernetes.io/projected/426ee25e-f4e1-44da-b11f-191d799f8f27-kube-api-access-zjt2l" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588242 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9271574f-8b74-4de7-8bcf-f90951a5fd4a" volumeName="kubernetes.io/configmap/9271574f-8b74-4de7-8bcf-f90951a5fd4a-nginx-conf" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588255 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaf3243-1eeb-4d94-bef9-524b67035ba2" volumeName="kubernetes.io/projected/deaf3243-1eeb-4d94-bef9-524b67035ba2-kube-api-access-sdr6s" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588273 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f98defbd-aa11-4a76-bac3-4d7aacdb44a1" volumeName="kubernetes.io/projected/f98defbd-aa11-4a76-bac3-4d7aacdb44a1-kube-api-access-7ggbz" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588301 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588315 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff626920-a094-4782-aae1-ac6bc9d0b359" volumeName="kubernetes.io/secret/ff626920-a094-4782-aae1-ac6bc9d0b359-serving-cert" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588330 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-registry-certificates" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588357 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="207974a1-670c-49f2-b0fb-3946d261e608" volumeName="kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588371 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bd90b6c-66f9-4ea8-8f73-2981acce43db" volumeName="kubernetes.io/empty-dir/7bd90b6c-66f9-4ea8-8f73-2981acce43db-data-volume" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588388 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb381a6-0d14-47a9-a144-4fd8be651947" volumeName="kubernetes.io/secret/0fb381a6-0d14-47a9-a144-4fd8be651947-serving-cert" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588403 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f73e2f3-6f98-41bd-abec-b72d60ac2bb5" volumeName="kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-stats-auth" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588417 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c9766ef-0c1a-4ddc-9918-e0c1fb68aada" volumeName="kubernetes.io/secret/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-agent-certs" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588430 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-image-registry-private-configuration" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588444 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77546eb8-8ef9-46b1-a47a-ca80ca5f76ee" volumeName="kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-config" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588458 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c9766ef-0c1a-4ddc-9918-e0c1fb68aada" volumeName="kubernetes.io/configmap/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-konnectivity-ca" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588474 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8faa206c-23f9-4f03-8b85-d66e47f08c30" volumeName="kubernetes.io/projected/8faa206c-23f9-4f03-8b85-d66e47f08c30-kube-api-access-lm8b7" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588489 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caf2ae77-8244-44fa-a47f-1b8555fed1f2" volumeName="kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-tuned" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588501 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588517 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb381a6-0d14-47a9-a144-4fd8be651947" volumeName="kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-service-ca-bundle" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588529 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb381a6-0d14-47a9-a144-4fd8be651947" volumeName="kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-snapshots" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588544 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31464cb4-534f-41a2-bc06-99343f47a433" volumeName="kubernetes.io/projected/31464cb4-534f-41a2-bc06-99343f47a433-kube-api-access-tm8lh" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588561 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674789e3-135f-4296-9de6-274e37588b04" volumeName="kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588575 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaf3243-1eeb-4d94-bef9-524b67035ba2" volumeName="kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-config" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588603 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588616 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="207974a1-670c-49f2-b0fb-3946d261e608" volumeName="kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588628 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36272f8-3fbc-4501-b03c-91a343e37204" volumeName="kubernetes.io/projected/a36272f8-3fbc-4501-b03c-91a343e37204-kube-api-access-j6b95" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588641 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f" volumeName="kubernetes.io/configmap/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-config" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588654 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaf3243-1eeb-4d94-bef9-524b67035ba2" volumeName="kubernetes.io/secret/deaf3243-1eeb-4d94-bef9-524b67035ba2-serving-cert" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588669 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f40f9013-d584-4546-940b-82f31a20cd4f" volumeName="kubernetes.io/projected/f40f9013-d584-4546-940b-82f31a20cd4f-kube-api-access-55k45" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588683 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a620cde-dc26-4159-8e40-41fdd982d61f" volumeName="kubernetes.io/projected/1a620cde-dc26-4159-8e40-41fdd982d61f-kube-api-access-ffnqz" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588700 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31464cb4-534f-41a2-bc06-99343f47a433" volumeName="kubernetes.io/empty-dir/31464cb4-534f-41a2-bc06-99343f47a433-tmp-dir" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588712 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97562254-d922-4415-9438-ae340b218942" volumeName="kubernetes.io/projected/97562254-d922-4415-9438-ae340b218942-kube-api-access-tn7xb" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588730 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97562254-d922-4415-9438-ae340b218942" volumeName="kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588744 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21" volumeName="kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cni-binary-copy" seLinuxMountContext="" Nov 02 19:48:21.588726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588756 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deaf3243-1eeb-4d94-bef9-524b67035ba2" volumeName="kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-trusted-ca" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588768 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588781 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-trusted-ca" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588793 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-registry-tls" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588808 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184c44a1-6433-4fb3-a7cd-ad9fa39cb642" volumeName="kubernetes.io/configmap/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-iptables-alerter-script" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588824 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184c44a1-6433-4fb3-a7cd-ad9fa39cb642" volumeName="kubernetes.io/projected/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-kube-api-access-zxznp" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588837 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703df830-74cd-46d0-96ab-ab6b2b42c439" volumeName="kubernetes.io/configmap/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-config" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588850 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77546eb8-8ef9-46b1-a47a-ca80ca5f76ee" volumeName="kubernetes.io/projected/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-kube-api-access-2z9vr" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588864 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9271574f-8b74-4de7-8bcf-f90951a5fd4a" volumeName="kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588876 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbaf72f9-c2ab-4081-87f4-7747ffc180ad" volumeName="kubernetes.io/configmap/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-cabundle" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588889 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a620cde-dc26-4159-8e40-41fdd982d61f" volumeName="kubernetes.io/configmap/1a620cde-dc26-4159-8e40-41fdd982d61f-config-volume" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588903 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b79f192-7b15-47d5-91b4-1772b3541ecf" volumeName="kubernetes.io/secret/3b79f192-7b15-47d5-91b4-1772b3541ecf-cert" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588915 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f73e2f3-6f98-41bd-abec-b72d60ac2bb5" volumeName="kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588928 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f73e2f3-6f98-41bd-abec-b72d60ac2bb5" volumeName="kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588942 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703df830-74cd-46d0-96ab-ab6b2b42c439" volumeName="kubernetes.io/projected/703df830-74cd-46d0-96ab-ab6b2b42c439-kube-api-access-jnjmr" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588955 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbaf72f9-c2ab-4081-87f4-7747ffc180ad" volumeName="kubernetes.io/secret/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-key" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588967 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f40f9013-d584-4546-940b-82f31a20cd4f" volumeName="kubernetes.io/configmap/f40f9013-d584-4546-940b-82f31a20cd4f-serviceca" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588983 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fea63025-526f-4fd8-b303-bc115c1226dd" volumeName="kubernetes.io/projected/fea63025-526f-4fd8-b303-bc115c1226dd-kube-api-access-mf4qd" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.588996 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f" volumeName="kubernetes.io/projected/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-kube-api-access-jhtw5" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589008 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21" volumeName="kubernetes.io/projected/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-kube-api-access-6xs6q" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589022 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5640e81-834c-4854-9cc9-589ccbd39d64" volumeName="kubernetes.io/projected/e5640e81-834c-4854-9cc9-589ccbd39d64-kube-api-access-s9m9j" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589034 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589046 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a63760e-f41b-4958-83a5-e4944090dccb" volumeName="kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-kube-api-access-8h788" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589064 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="505ade63-4db5-4709-9e6a-cdaed488b662" volumeName="kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589077 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="674789e3-135f-4296-9de6-274e37588b04" volumeName="kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589094 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77546eb8-8ef9-46b1-a47a-ca80ca5f76ee" volumeName="kubernetes.io/secret/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589108 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caf2ae77-8244-44fa-a47f-1b8555fed1f2" volumeName="kubernetes.io/projected/caf2ae77-8244-44fa-a47f-1b8555fed1f2-kube-api-access-qpj8m" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589123 5800 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe5633bb-4256-4685-943e-0e0ecbce30cd" volumeName="kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq" seLinuxMountContext="" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589135 5800 reconstruct.go:97] "Volume reconstruction finished" Nov 02 19:48:21.590835 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.589143 5800 reconciler.go:29] "Reconciler: start to sync state" Nov 02 19:48:21.601142 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.601120 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:48:21.603559 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.603540 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:48:21.612436 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.611790 5800 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 19:48:21.614397 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.613720 5800 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 19:48:21.614397 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.613745 5800 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 19:48:21.614397 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.613770 5800 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 19:48:21.614397 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.613812 5800 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 19:48:21.672428 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.672356 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:48:21.672428 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.672393 5800 generic.go:356] "Generic (PLEG): container finished" podID="deaf3243-1eeb-4d94-bef9-524b67035ba2" containerID="af3dc24e1e3b99eeaabf84378cdedff256741f25d6a0946fadd6d653d68e71c2" exitCode=255 Nov 02 19:48:21.675718 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.675688 5800 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="c5bacd6bdee74d38592f0e1f2a8a51a243236096cfc6af0f4a67f86af51a9bfe" exitCode=0 Nov 02 19:48:21.675833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.675719 5800 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="1786d35d12092f97eafaad66da7959b09c246b0fbd3164893951215157b93c8c" exitCode=0 Nov 02 19:48:21.675833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.675731 5800 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="5616231cbaa0c19d40414acc403b2fae3bf6f87ce7eb7a2d5602cfe65358301c" exitCode=0 Nov 02 19:48:21.675833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.675744 5800 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="5eae745bd2f0c80b122a486a8e39821e45fca1c7aea2f4b36eba67cff5c13234" exitCode=0 Nov 02 19:48:21.675833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.675755 5800 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="c6882e11c0ca8ca453556a16e765d91f344c167f124b056a3919b6ba86a51fc7" exitCode=0 Nov 02 19:48:21.675833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.675763 5800 generic.go:356] "Generic (PLEG): container finished" podID="207974a1-670c-49f2-b0fb-3946d261e608" containerID="c8f73ca55198688762ef07593815697c8675a62c972af43f4e709adb9b7820f3" exitCode=0 Nov 02 19:48:21.678944 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.678923 5800 generic.go:356] "Generic (PLEG): container finished" podID="68d4a7ab7ffda58b80be6a631c2486f0" containerID="83ec6f649c39f602c775a78b3b77b499c34977895f5d6309a0b978b4a9827d53" exitCode=0 Nov 02 19:48:21.683517 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.683494 5800 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:48:21.702372 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.702354 5800 manager.go:324] Recovery completed Nov 02 19:48:21.714271 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.714247 5800 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 19:48:21.725583 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.725564 5800 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:21.728318 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.728301 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:21.728414 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.728336 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:21.728414 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.728350 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:21.731250 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731226 5800 cpu_manager.go:226] "Starting" policy="none" Nov 02 19:48:21.731250 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731243 5800 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:21.731435 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731246 5800 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 19:48:21.731435 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731315 5800 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 19:48:21.731510 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731449 5800 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 19:48:21.731510 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731460 5800 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 19:48:21.731510 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731477 5800 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 19:48:21.731510 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731483 5800 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 19:48:21.731510 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731488 5800 policy_none.go:50] "Start" Nov 02 19:48:21.731510 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731497 5800 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 19:48:21.731510 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731507 5800 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 19:48:21.731692 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731629 5800 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 19:48:21.731692 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731637 5800 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 19:48:21.731692 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.731643 5800 policy_none.go:44] "Start" Nov 02 19:48:21.732233 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.732211 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:21.732345 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.732246 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:21.732345 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.732256 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:21.736733 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.736706 5800 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 19:48:21.739325 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739310 5800 manager.go:341] "Starting Device Plugin manager" Nov 02 19:48:21.739403 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.739335 5800 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 19:48:21.739403 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739346 5800 server.go:87] "Starting device plugin registration server" Nov 02 19:48:21.739617 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739605 5800 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 19:48:21.739669 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739616 5800 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 19:48:21.739734 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739713 5800 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 19:48:21.739878 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739835 5800 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 19:48:21.739878 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739886 5800 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 19:48:21.739987 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739893 5800 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 19:48:21.740038 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.739938 5800 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T19:48:21.739876028Z","UUID":"a2aadacb-1cd2-4369-b5a9-3c3baae67967","Handler":null,"Name":"","Endpoint":""} Nov 02 19:48:21.741350 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.741330 5800 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 19:48:21.741421 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.741366 5800 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-13.us-west-2.compute.internal\" not found" Nov 02 19:48:21.741753 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.741736 5800 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 19:48:21.741827 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.741761 5800 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 19:48:21.766125 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.766097 5800 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:21.766486 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.766471 5800 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:21.766590 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.766577 5800 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 19:48:21.766924 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.766909 5800 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:21.813468 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.813443 5800 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 19:48:21.840524 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.840502 5800 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 19:48:21.841604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.841589 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 19:48:21.841685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.841616 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 19:48:21.841685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.841626 5800 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-13.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 19:48:21.841972 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.841955 5800 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:21.914878 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.914833 5800 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal"] Nov 02 19:48:21.915528 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.915495 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"7161e7f2a8cfc7fa3ac048d9b7921179","Type":"ContainerStarted","Data":"5eb6979ad7a74feec2c4f8db83e9fcfc1cf1a4f79d03dd5a07ef379ba99faf2e"} Nov 02 19:48:21.915595 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.915529 5800 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:21.915629 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.915534 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"7161e7f2a8cfc7fa3ac048d9b7921179","Type":"ContainerStarted","Data":"5b45ca5342deaff5cad00314f44d48cad7482f7971e1eb7c2afb4de4a93df4c8"} Nov 02 19:48:21.915704 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.915692 5800 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:21.915787 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.915763 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"68d4a7ab7ffda58b80be6a631c2486f0","Type":"ContainerStarted","Data":"42c3576ba5fee9e633296f5a759ede3371e925f72b0773b067ab8b671131947e"} Nov 02 19:48:21.915819 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.915799 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"68d4a7ab7ffda58b80be6a631c2486f0","Type":"ContainerDied","Data":"83ec6f649c39f602c775a78b3b77b499c34977895f5d6309a0b978b4a9827d53"} Nov 02 19:48:21.915819 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.915815 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" event={"ID":"68d4a7ab7ffda58b80be6a631c2486f0","Type":"ContainerStarted","Data":"fa50067078ba89423e5d0fd1b3840b71e6f3283e70a0b25e89d7decda8d621ff"} Nov 02 19:48:21.959769 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.959699 5800 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:21.959769 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.959770 5800 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:21.973368 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.973345 5800 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:48:21.973495 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.973394 5800 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:21.974077 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:21.974062 5800 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:48:21.974119 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:21.974104 5800 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.067465 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.067430 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.067465 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.067465 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.067678 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.067483 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7161e7f2a8cfc7fa3ac048d9b7921179-config\") pod \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"7161e7f2a8cfc7fa3ac048d9b7921179\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.168663 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.168624 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.168663 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.168664 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.168841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.168683 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7161e7f2a8cfc7fa3ac048d9b7921179-config\") pod \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"7161e7f2a8cfc7fa3ac048d9b7921179\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.168841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.168729 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7161e7f2a8cfc7fa3ac048d9b7921179-config\") pod \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"7161e7f2a8cfc7fa3ac048d9b7921179\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.168841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.168730 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.168841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.168730 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68d4a7ab7ffda58b80be6a631c2486f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" (UID: \"68d4a7ab7ffda58b80be6a631c2486f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.568387 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.568347 5800 apiserver.go:52] "Watching apiserver" Nov 02 19:48:22.685334 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.685303 5800 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.685519 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.685424 5800 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.711866 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.711831 5800 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 19:48:22.712689 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.712654 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kg2ng","openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc","openshift-console-operator/console-operator-7cd56b8dbc-2b245","openshift-cluster-node-tuning-operator/tuned-vj7jb","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv","kube-system/kas-connection-checker-685895c7c9-6s4x6","openshift-ingress-canary/ingress-canary-dkb72","openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt","openshift-service-ca/service-ca-5fffc97bdf-cwvdn","openshift-dns/dns-default-27pq5","openshift-image-registry/image-registry-d458f586f-ndk5f","openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9","openshift-image-registry/node-ca-pcsq4","openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7","kube-system/kas-connection-checker-685895c7c9-t25fz","openshift-dns/node-resolver-5bq8f","openshift-image-registry/image-registry-5bffd989f9-r4bss","openshift-multus/multus-5slm4","openshift-network-console/networking-console-plugin-86c478bb88-pxnkj","openshift-ingress/router-default-5f6668f8f7-2vjcr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4","kube-system/konnectivity-agent-trg79","openshift-multus/network-metrics-daemon-nt26r","openshift-network-operator/iptables-alerter-nm6dq","kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-8sfkw","kube-system/global-pull-secret-syncer-j7kp4","openshift-network-diagnostics/network-check-target-mpxzs","openshift-insights/insights-runtime-extractor-x6hpx","kube-system/kas-connection-checker-685895c7c9-qwntm","openshift-insights/periodic-gathering-gcdjm-447t7","openshift-multus/multus-additional-cni-plugins-pq2jl"] Nov 02 19:48:22.728526 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.728496 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.741624 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.741594 5800 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:48:22.741758 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:22.741648 5800 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.742148 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.742133 5800 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 19:48:22.742184 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:22.742170 5800 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-13.us-west-2.compute.internal" Nov 02 19:48:22.764905 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.764874 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 19:48:22.765310 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765267 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 19:48:22.765411 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765392 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 19:48:22.765467 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765452 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.765516 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765480 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 19:48:22.765563 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765534 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.765563 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765546 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 19:48:22.765652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765604 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 19:48:22.765652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765607 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q7g7q\"" Nov 02 19:48:22.765652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765607 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 19:48:22.765652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765642 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765644 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765667 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765701 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765702 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765740 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-t6dtl\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765747 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765741 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765763 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765751 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765831 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.765841 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765846 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pwftm\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765881 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765982 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mg6c7\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.765993 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766011 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766030 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766047 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h9lqd\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766067 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766073 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 19:48:22.766370 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766350 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.766622 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766376 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 19:48:22.766678 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766657 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.766755 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766737 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9xjzz\"" Nov 02 19:48:22.766969 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766951 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vvz96\"" Nov 02 19:48:22.767008 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.766986 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6n7fh\"" Nov 02 19:48:22.767041 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.767005 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 19:48:22.767041 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.767034 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 19:48:22.767723 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.767709 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 19:48:22.786393 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.786369 5800 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 19:48:22.811978 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.811946 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 19:48:22.815100 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.815073 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 19:48:22.815904 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.815879 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 19:48:22.816063 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816048 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 19:48:22.816297 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816264 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 19:48:22.816403 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816387 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.816475 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816403 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.816570 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816549 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 19:48:22.816631 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816615 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.816631 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816390 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 19:48:22.816724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816490 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-t7wfl\"" Nov 02 19:48:22.816724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816508 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7cqmh\"" Nov 02 19:48:22.816724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816522 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 19:48:22.816724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816655 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.816724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816696 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.816724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816716 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.816995 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816536 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fhf7l\"" Nov 02 19:48:22.816995 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816746 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.816995 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816861 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.816995 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816882 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2gbh4\"" Nov 02 19:48:22.816995 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816951 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817000 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817020 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817050 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5cssh\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816885 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xk7lc\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817087 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817117 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817072 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p97wz\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817181 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rmffj\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817194 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kkdt\"" Nov 02 19:48:22.817232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817234 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dsl5q\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817244 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817262 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817264 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817331 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wj4w6\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817350 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817355 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817379 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gkplk\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817023 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.816269 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817428 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817489 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817506 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817508 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 19:48:22.817675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.817619 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818009 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818105 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7q2qk\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818119 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818128 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818120 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qzg2s\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818141 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818168 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 19:48:22.818173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818173 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818197 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tnfsn\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818209 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818247 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818260 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818272 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818315 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818399 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818437 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818449 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818459 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 19:48:22.818557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818531 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6t9zq\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818598 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818607 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818655 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818663 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818705 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-z7kgr\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818733 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-869qp\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818751 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7zhgs\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818772 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818894 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9wbn\"" Nov 02 19:48:22.819023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.818919 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 19:48:22.857572 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.857545 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 19:48:22.872001 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.871973 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-host-slash\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:48:22.872070 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872012 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggbz\" (UniqueName: \"kubernetes.io/projected/f98defbd-aa11-4a76-bac3-4d7aacdb44a1-kube-api-access-7ggbz\") pod \"kas-connection-checker-685895c7c9-t25fz\" (UID: \"f98defbd-aa11-4a76-bac3-4d7aacdb44a1\") " pod="kube-system/kas-connection-checker-685895c7c9-t25fz" Nov 02 19:48:22.872070 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872030 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47cn\" (UniqueName: \"kubernetes.io/projected/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-kube-api-access-r47cn\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.872070 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872047 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:48:22.872202 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872103 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-binary-copy\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.872202 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872181 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.872326 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872222 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.872326 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872256 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct6x2\" (UniqueName: \"kubernetes.io/projected/ff626920-a094-4782-aae1-ac6bc9d0b359-kube-api-access-ct6x2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:48:22.872326 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872306 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.872473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872351 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8b7\" (UniqueName: \"kubernetes.io/projected/8faa206c-23f9-4f03-8b85-d66e47f08c30-kube-api-access-lm8b7\") pod \"network-check-source-6f78d59f99-95m8f\" (UID: \"8faa206c-23f9-4f03-8b85-d66e47f08c30\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" Nov 02 19:48:22.872473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872360 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-binary-copy\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.872473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872383 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-etc-kubernetes\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.872473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872419 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-modprobe-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.872473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872382 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9271574f-8b74-4de7-8bcf-f90951a5fd4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:48:22.872473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872455 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-tmp\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.872473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872464 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872488 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxznp\" (UniqueName: \"kubernetes.io/projected/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-kube-api-access-zxznp\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872516 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-systemd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872537 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-tmp\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872543 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsfs\" (UniqueName: \"kubernetes.io/projected/0fb381a6-0d14-47a9-a144-4fd8be651947-kube-api-access-wfsfs\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872591 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-kubelet\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872630 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckjq\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872634 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872671 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-host\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.872779 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872773 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-etc-selinux\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872820 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872852 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-slash\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872880 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.872965 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873020 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873033 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873078 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.873124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873121 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873147 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873181 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873187 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873219 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-dbus\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873225 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-service-ca-bundle\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873332 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-agent-certs\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873369 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a620cde-dc26-4159-8e40-41fdd982d61f-config-volume\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.873490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873477 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnnd\" (UniqueName: \"kubernetes.io/projected/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-api-access-rnnnd\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873533 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a620cde-dc26-4159-8e40-41fdd982d61f-config-volume\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873534 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873635 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873646 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-agent-certs\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873654 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-systemd-units\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873670 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/207974a1-670c-49f2-b0fb-3946d261e608-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873679 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fb381a6-0d14-47a9-a144-4fd8be651947-serving-cert\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873696 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873716 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-multus\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.873770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873733 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873790 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873828 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcnh\" (UniqueName: \"kubernetes.io/projected/cb5fc542-006a-4da5-b985-cc7b0f1709a7-kube-api-access-8mcnh\") pod \"kas-connection-checker-685895c7c9-qwntm\" (UID: \"cb5fc542-006a-4da5-b985-cc7b0f1709a7\") " pod="kube-system/kas-connection-checker-685895c7c9-qwntm" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873850 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fea63025-526f-4fd8-b303-bc115c1226dd-metrics-certs\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873854 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-systemd\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873881 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-stats-auth\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873908 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-system-cni-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873951 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f40f9013-d584-4546-940b-82f31a20cd4f-serviceca\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.873999 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874056 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-node-log\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874127 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874162 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cni-binary-copy\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.874194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874176 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874200 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-stats-auth\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874212 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-bound-sa-token\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874253 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0fb381a6-0d14-47a9-a144-4fd8be651947-serving-cert\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874259 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-default-certificate\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874328 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31464cb4-534f-41a2-bc06-99343f47a433-tmp-dir\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874350 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cni-binary-copy\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874358 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9m9j\" (UniqueName: \"kubernetes.io/projected/e5640e81-834c-4854-9cc9-589ccbd39d64-kube-api-access-s9m9j\") pod \"kas-connection-checker-685895c7c9-6s4x6\" (UID: \"e5640e81-834c-4854-9cc9-589ccbd39d64\") " pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874421 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31464cb4-534f-41a2-bc06-99343f47a433-tmp-dir\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874467 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4qd\" (UniqueName: \"kubernetes.io/projected/fea63025-526f-4fd8-b303-bc115c1226dd-kube-api-access-mf4qd\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874540 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-default-certificate\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874567 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovn-node-metrics-cert\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874617 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f40f9013-d584-4546-940b-82f31a20cd4f-serviceca\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:48:22.874645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874650 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-key\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874721 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-socket-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874806 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874849 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874883 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovn-node-metrics-cert\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874883 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-signing-key\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874928 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-netns\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.874959 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875021 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-registry-certificates\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875049 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-run\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875074 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-tuned\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875099 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875114 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eceae17b-5b0b-45ce-87cf-9d720583c247-original-pull-secret\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875137 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-trusted-ca\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.875188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875169 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-tuned\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875202 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875264 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875313 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-registry-certificates\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875321 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875380 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-socket-dir-parent\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875450 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/703df830-74cd-46d0-96ab-ab6b2b42c439-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875481 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875500 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-trusted-ca\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875521 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxmg\" (UniqueName: \"kubernetes.io/projected/3b79f192-7b15-47d5-91b4-1772b3541ecf-kube-api-access-6cxmg\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875557 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875593 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97562254-d922-4415-9438-ae340b218942-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875624 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-kubelet-config\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875706 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt2l\" (UniqueName: \"kubernetes.io/projected/426ee25e-f4e1-44da-b11f-191d799f8f27-kube-api-access-zjt2l\") pod \"migrator-66df5d496b-z2znt\" (UID: \"426ee25e-f4e1-44da-b11f-191d799f8f27\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875724 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn7xb\" (UniqueName: \"kubernetes.io/projected/97562254-d922-4415-9438-ae340b218942-kube-api-access-tn7xb\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875739 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-log-socket\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875755 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-netns\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875772 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-image-registry-private-configuration\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875801 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h788\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-kube-api-access-8h788\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875821 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-conf\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.875833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875835 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-sys\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875851 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-config\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875892 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.875937 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-image-registry-private-configuration\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876019 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/deaf3243-1eeb-4d94-bef9-524b67035ba2-config\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876025 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cnibin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876115 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-script-lib\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876142 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-bin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876166 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-hostroot\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876194 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-metrics-certs\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876253 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-device-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876271 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdr6s\" (UniqueName: \"kubernetes.io/projected/deaf3243-1eeb-4d94-bef9-524b67035ba2-kube-api-access-sdr6s\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876322 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bd90b6c-66f9-4ea8-8f73-2981acce43db-crio-socket\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876330 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-script-lib\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876355 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876387 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876421 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-installation-pull-secrets\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876448 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31464cb4-534f-41a2-bc06-99343f47a433-hosts-file\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:48:22.876513 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876469 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm8lh\" (UniqueName: \"kubernetes.io/projected/31464cb4-534f-41a2-bc06-99343f47a433-kube-api-access-tm8lh\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876638 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-tmp\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876674 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876698 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a63760e-f41b-4958-83a5-e4944090dccb-installation-pull-secrets\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876731 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876787 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-tmp\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876811 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-konnectivity-ca\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876829 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f40f9013-d584-4546-940b-82f31a20cd4f-host\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876845 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-system-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876861 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b79f192-7b15-47d5-91b4-1772b3541ecf-cert\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876877 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjmr\" (UniqueName: \"kubernetes.io/projected/703df830-74cd-46d0-96ab-ab6b2b42c439-kube-api-access-jnjmr\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:48:22.876996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.876909 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-etc-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877004 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff626920-a094-4782-aae1-ac6bc9d0b359-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877014 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c9766ef-0c1a-4ddc-9918-e0c1fb68aada-konnectivity-ca\") pod \"konnectivity-agent-trg79\" (UID: \"7c9766ef-0c1a-4ddc-9918-e0c1fb68aada\") " pod="kube-system/konnectivity-agent-trg79" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877058 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b79f192-7b15-47d5-91b4-1772b3541ecf-cert\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877062 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xs6q\" (UniqueName: \"kubernetes.io/projected/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-kube-api-access-6xs6q\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877095 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-sys-fs\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877162 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-snapshots\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877228 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-registry-tls\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877321 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtw5\" (UniqueName: \"kubernetes.io/projected/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-kube-api-access-jhtw5\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:48:22.877348 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877338 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0fb381a6-0d14-47a9-a144-4fd8be651947-snapshots\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877352 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-iptables-alerter-script\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877353 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff626920-a094-4782-aae1-ac6bc9d0b359-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877423 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-registry-tls\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877467 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-kubelet\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877493 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-env-overrides\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877519 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-daemon-config\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877532 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-iptables-alerter-script\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877549 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-multus-certs\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877567 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-config\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877584 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff626920-a094-4782-aae1-ac6bc9d0b359-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877600 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-k8s-cni-cncf-io\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.877604 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877606 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-env-overrides\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877615 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deaf3243-1eeb-4d94-bef9-524b67035ba2-serving-cert\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877638 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnqz\" (UniqueName: \"kubernetes.io/projected/1a620cde-dc26-4159-8e40-41fdd982d61f-kube-api-access-ffnqz\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877674 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-bin\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877691 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-daemon-config\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877706 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-config\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877743 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9vr\" (UniqueName: \"kubernetes.io/projected/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-kube-api-access-2z9vr\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877755 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff626920-a094-4782-aae1-ac6bc9d0b359-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877788 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-config\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877801 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877868 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/deaf3243-1eeb-4d94-bef9-524b67035ba2-serving-cert\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877927 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-ovnkube-config\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877938 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a63760e-f41b-4958-83a5-e4944090dccb-ca-trust-extracted\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877969 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-lib-modules\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877995 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.878044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.877997 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-var-lib-kubelet\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878053 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-ovn\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878067 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a63760e-f41b-4958-83a5-e4944090dccb-ca-trust-extracted\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878115 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878143 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-service-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878159 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj8m\" (UniqueName: \"kubernetes.io/projected/caf2ae77-8244-44fa-a47f-1b8555fed1f2-kube-api-access-qpj8m\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878178 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-registration-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878206 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdkl\" (UniqueName: \"kubernetes.io/projected/4b70b3c0-c849-4af0-904f-677aa9222a1e-kube-api-access-2xdkl\") pod \"migrator-66df5d496b-9gfq4\" (UID: \"4b70b3c0-c849-4af0-904f-677aa9222a1e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878232 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55k45\" (UniqueName: \"kubernetes.io/projected/f40f9013-d584-4546-940b-82f31a20cd4f-kube-api-access-55k45\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878258 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b95\" (UniqueName: \"kubernetes.io/projected/a36272f8-3fbc-4501-b03c-91a343e37204-kube-api-access-j6b95\") pod \"volume-data-source-validator-58c9f5f46d-55rd9\" (UID: \"a36272f8-3fbc-4501-b03c-91a343e37204\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878402 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878421 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-serving-cert\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878438 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwfw\" (UniqueName: \"kubernetes.io/projected/207974a1-670c-49f2-b0fb-3946d261e608-kube-api-access-jkwfw\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878461 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-netd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878491 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878516 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysconfig\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878544 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-kubernetes\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878570 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bd90b6c-66f9-4ea8-8f73-2981acce43db-data-volume\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.878603 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878596 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9271574f-8b74-4de7-8bcf-f90951a5fd4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878617 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-conf-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878643 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a620cde-dc26-4159-8e40-41fdd982d61f-tmp-dir\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878669 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97562254-d922-4415-9438-ae340b218942-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878682 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a620cde-dc26-4159-8e40-41fdd982d61f-metrics-tls\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878697 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-var-lib-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878704 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-service-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878722 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bd90b6c-66f9-4ea8-8f73-2981acce43db-data-volume\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878726 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-trusted-ca\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878727 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fb381a6-0d14-47a9-a144-4fd8be651947-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878782 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-serving-cert\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878782 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a620cde-dc26-4159-8e40-41fdd982d61f-tmp-dir\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878793 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh5c\" (UniqueName: \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878829 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-cnibin\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878931 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9271574f-8b74-4de7-8bcf-f90951a5fd4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-pxnkj\" (UID: \"9271574f-8b74-4de7-8bcf-f90951a5fd4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-pxnkj" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878942 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6txz\" (UniqueName: \"kubernetes.io/projected/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-kube-api-access-r6txz\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878964 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqs4h\" (UniqueName: \"kubernetes.io/projected/5e413f92-3561-4f46-a45b-1fbe4919af71-kube-api-access-hqs4h\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878964 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97562254-d922-4415-9438-ae340b218942-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.878985 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.879335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.879173 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a63760e-f41b-4958-83a5-e4944090dccb-trusted-ca\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.879823 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.879663 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.927512 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.927461 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct6x2\" (UniqueName: \"kubernetes.io/projected/ff626920-a094-4782-aae1-ac6bc9d0b359-kube-api-access-ct6x2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc\" (UID: \"ff626920-a094-4782-aae1-ac6bc9d0b359\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" Nov 02 19:48:22.927659 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.927639 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsfs\" (UniqueName: \"kubernetes.io/projected/0fb381a6-0d14-47a9-a144-4fd8be651947-kube-api-access-wfsfs\") pod \"insights-operator-7c4986d449-kg2ng\" (UID: \"0fb381a6-0d14-47a9-a144-4fd8be651947\") " pod="openshift-insights/insights-operator-7c4986d449-kg2ng" Nov 02 19:48:22.927733 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.927706 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.927769 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.927724 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8b7\" (UniqueName: \"kubernetes.io/projected/8faa206c-23f9-4f03-8b85-d66e47f08c30-kube-api-access-lm8b7\") pod \"network-check-source-6f78d59f99-95m8f\" (UID: \"8faa206c-23f9-4f03-8b85-d66e47f08c30\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-95m8f" Nov 02 19:48:22.927769 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.927736 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47cn\" (UniqueName: \"kubernetes.io/projected/6f73e2f3-6f98-41bd-abec-b72d60ac2bb5-kube-api-access-r47cn\") pod \"router-default-5f6668f8f7-2vjcr\" (UID: \"6f73e2f3-6f98-41bd-abec-b72d60ac2bb5\") " pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:22.927826 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.927792 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxznp\" (UniqueName: \"kubernetes.io/projected/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-kube-api-access-zxznp\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:48:22.927913 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.927891 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckjq\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\") pod \"image-registry-5bffd989f9-r4bss\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:22.928443 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.928422 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggbz\" (UniqueName: \"kubernetes.io/projected/f98defbd-aa11-4a76-bac3-4d7aacdb44a1-kube-api-access-7ggbz\") pod \"kas-connection-checker-685895c7c9-t25fz\" (UID: \"f98defbd-aa11-4a76-bac3-4d7aacdb44a1\") " pod="kube-system/kas-connection-checker-685895c7c9-t25fz" Nov 02 19:48:22.930690 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.930673 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-bound-sa-token\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.931546 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.931531 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt2l\" (UniqueName: \"kubernetes.io/projected/426ee25e-f4e1-44da-b11f-191d799f8f27-kube-api-access-zjt2l\") pod \"migrator-66df5d496b-z2znt\" (UID: \"426ee25e-f4e1-44da-b11f-191d799f8f27\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-z2znt" Nov 02 19:48:22.931667 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.931651 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcnh\" (UniqueName: \"kubernetes.io/projected/cb5fc542-006a-4da5-b985-cc7b0f1709a7-kube-api-access-8mcnh\") pod \"kas-connection-checker-685895c7c9-qwntm\" (UID: \"cb5fc542-006a-4da5-b985-cc7b0f1709a7\") " pod="kube-system/kas-connection-checker-685895c7c9-qwntm" Nov 02 19:48:22.932447 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932425 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h788\" (UniqueName: \"kubernetes.io/projected/0a63760e-f41b-4958-83a5-e4944090dccb-kube-api-access-8h788\") pod \"image-registry-d458f586f-ndk5f\" (UID: \"0a63760e-f41b-4958-83a5-e4944090dccb\") " pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:22.932554 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932491 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnnd\" (UniqueName: \"kubernetes.io/projected/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-api-access-rnnnd\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.932554 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932522 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9m9j\" (UniqueName: \"kubernetes.io/projected/e5640e81-834c-4854-9cc9-589ccbd39d64-kube-api-access-s9m9j\") pod \"kas-connection-checker-685895c7c9-6s4x6\" (UID: \"e5640e81-834c-4854-9cc9-589ccbd39d64\") " pod="kube-system/kas-connection-checker-685895c7c9-6s4x6" Nov 02 19:48:22.932682 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932556 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtw5\" (UniqueName: \"kubernetes.io/projected/b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f-kube-api-access-jhtw5\") pod \"service-ca-operator-544d749c8c-5x8qv\" (UID: \"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" Nov 02 19:48:22.932682 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932637 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxmg\" (UniqueName: \"kubernetes.io/projected/3b79f192-7b15-47d5-91b4-1772b3541ecf-kube-api-access-6cxmg\") pod \"ingress-canary-dkb72\" (UID: \"3b79f192-7b15-47d5-91b4-1772b3541ecf\") " pod="openshift-ingress-canary/ingress-canary-dkb72" Nov 02 19:48:22.932800 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932768 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn7xb\" (UniqueName: \"kubernetes.io/projected/97562254-d922-4415-9438-ae340b218942-kube-api-access-tn7xb\") pod \"cluster-monitoring-operator-884576c57-kz9f7\" (UID: \"97562254-d922-4415-9438-ae340b218942\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kz9f7" Nov 02 19:48:22.932860 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932801 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdr6s\" (UniqueName: \"kubernetes.io/projected/deaf3243-1eeb-4d94-bef9-524b67035ba2-kube-api-access-sdr6s\") pod \"console-operator-7cd56b8dbc-2b245\" (UID: \"deaf3243-1eeb-4d94-bef9-524b67035ba2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:22.932976 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932957 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnqz\" (UniqueName: \"kubernetes.io/projected/1a620cde-dc26-4159-8e40-41fdd982d61f-kube-api-access-ffnqz\") pod \"dns-default-27pq5\" (UID: \"1a620cde-dc26-4159-8e40-41fdd982d61f\") " pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:22.933023 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.932983 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4qd\" (UniqueName: \"kubernetes.io/projected/fea63025-526f-4fd8-b303-bc115c1226dd-kube-api-access-mf4qd\") pod \"network-metrics-daemon-nt26r\" (UID: \"fea63025-526f-4fd8-b303-bc115c1226dd\") " pod="openshift-multus/network-metrics-daemon-nt26r" Nov 02 19:48:22.933187 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.933167 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xs6q\" (UniqueName: \"kubernetes.io/projected/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-kube-api-access-6xs6q\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.933311 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.933294 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2hd\" (UniqueName: \"kubernetes.io/projected/505ade63-4db5-4709-9e6a-cdaed488b662-kube-api-access-gc2hd\") pod \"network-check-target-mpxzs\" (UID: \"505ade63-4db5-4709-9e6a-cdaed488b662\") " pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:22.933367 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.933314 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjmr\" (UniqueName: \"kubernetes.io/projected/703df830-74cd-46d0-96ab-ab6b2b42c439-kube-api-access-jnjmr\") pod \"cluster-samples-operator-6ddb6c8b94-wvrhv\" (UID: \"703df830-74cd-46d0-96ab-ab6b2b42c439\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wvrhv" Nov 02 19:48:22.933632 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.933611 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqs4h\" (UniqueName: \"kubernetes.io/projected/5e413f92-3561-4f46-a45b-1fbe4919af71-kube-api-access-hqs4h\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.933792 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.933773 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55k45\" (UniqueName: \"kubernetes.io/projected/f40f9013-d584-4546-940b-82f31a20cd4f-kube-api-access-55k45\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:48:22.933862 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.933846 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6txz\" (UniqueName: \"kubernetes.io/projected/dbaf72f9-c2ab-4081-87f4-7747ffc180ad-kube-api-access-r6txz\") pod \"service-ca-5fffc97bdf-cwvdn\" (UID: \"dbaf72f9-c2ab-4081-87f4-7747ffc180ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cwvdn" Nov 02 19:48:22.934141 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.934119 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9vr\" (UniqueName: \"kubernetes.io/projected/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-kube-api-access-2z9vr\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.934202 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.934155 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdkl\" (UniqueName: \"kubernetes.io/projected/4b70b3c0-c849-4af0-904f-677aa9222a1e-kube-api-access-2xdkl\") pod \"migrator-66df5d496b-9gfq4\" (UID: \"4b70b3c0-c849-4af0-904f-677aa9222a1e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-9gfq4" Nov 02 19:48:22.934311 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.934269 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwfw\" (UniqueName: \"kubernetes.io/projected/207974a1-670c-49f2-b0fb-3946d261e608-kube-api-access-jkwfw\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.934311 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.934297 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b95\" (UniqueName: \"kubernetes.io/projected/a36272f8-3fbc-4501-b03c-91a343e37204-kube-api-access-j6b95\") pod \"volume-data-source-validator-58c9f5f46d-55rd9\" (UID: \"a36272f8-3fbc-4501-b03c-91a343e37204\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-55rd9" Nov 02 19:48:22.934690 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.934674 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm8lh\" (UniqueName: \"kubernetes.io/projected/31464cb4-534f-41a2-bc06-99343f47a433-kube-api-access-tm8lh\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:48:22.934732 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.934690 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj8m\" (UniqueName: \"kubernetes.io/projected/caf2ae77-8244-44fa-a47f-1b8555fed1f2-kube-api-access-qpj8m\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.934967 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.934952 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh5c\" (UniqueName: \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\") pod \"periodic-gathering-gcdjm-447t7\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:22.980108 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980065 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-kubelet-config\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.980239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980117 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-log-socket\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.980239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980155 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-netns\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980164 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-log-socket\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.980239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980182 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-conf\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.980239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980200 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-sys\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.980239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980214 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-kubelet-config\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.980239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980215 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cnibin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980237 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-netns\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980252 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-sys\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980243 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-bin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980263 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-bin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980302 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-hostroot\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980307 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-cnibin\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980328 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-device-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980338 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-hostroot\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980356 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bd90b6c-66f9-4ea8-8f73-2981acce43db-crio-socket\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980375 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-conf\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980391 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-device-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980380 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980421 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7bd90b6c-66f9-4ea8-8f73-2981acce43db-crio-socket\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980428 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980451 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31464cb4-534f-41a2-bc06-99343f47a433-hosts-file\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980473 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f40f9013-d584-4546-940b-82f31a20cd4f-host\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980497 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980497 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-system-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980504 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysctl-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980511 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31464cb4-534f-41a2-bc06-99343f47a433-hosts-file\") pod \"node-resolver-5bq8f\" (UID: \"31464cb4-534f-41a2-bc06-99343f47a433\") " pod="openshift-dns/node-resolver-5bq8f" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980524 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-etc-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980530 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f40f9013-d584-4546-940b-82f31a20cd4f-host\") pod \"node-ca-pcsq4\" (UID: \"f40f9013-d584-4546-940b-82f31a20cd4f\") " pod="openshift-image-registry/node-ca-pcsq4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980546 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-sys-fs\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980549 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-system-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980559 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-etc-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980581 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-kubelet\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980589 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-sys-fs\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.980582 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980600 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-multus-certs\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980618 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-k8s-cni-cncf-io\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980623 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-kubelet\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980640 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-bin\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980664 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-multus-certs\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980677 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-run-k8s-cni-cncf-io\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980682 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-bin\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980692 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-lib-modules\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980714 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-var-lib-kubelet\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980730 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-ovn\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980745 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980769 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-registration-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980782 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-var-lib-kubelet\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980791 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-ovn\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980799 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-netd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980794 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-lib-modules\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980807 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980822 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysconfig\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980833 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-registration-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980840 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-cni-netd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980845 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-kubernetes\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980868 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-conf-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980884 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-var-lib-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980896 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-sysconfig\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980905 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-kubernetes\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980915 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-var-lib-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980910 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-conf-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980921 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-cnibin\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980941 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-host-slash\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980945 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-cnibin\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980958 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980975 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-etc-kubernetes\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.980981 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/184c44a1-6433-4fb3-a7cd-ad9fa39cb642-host-slash\") pod \"iptables-alerter-nm6dq\" (UID: \"184c44a1-6433-4fb3-a7cd-ad9fa39cb642\") " pod="openshift-network-operator/iptables-alerter-nm6dq" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981001 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-etc-kubernetes\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981011 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-openvswitch\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981045 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-modprobe-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981072 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-systemd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981098 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-kubelet\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981122 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-host\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981142 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-modprobe-d\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981149 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-etc-selinux\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981153 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-kubelet\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981177 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-host\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981178 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981145 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-run-systemd\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981196 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-etc-selinux\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981202 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-slash\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981254 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981270 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-slash\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981302 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-dbus\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981332 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-systemd-units\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981357 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981381 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-multus\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981400 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-systemd-units\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981408 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-systemd\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981431 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-host-var-lib-cni-multus\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981436 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-system-cni-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981442 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eceae17b-5b0b-45ce-87cf-9d720583c247-dbus\") pod \"global-pull-secret-syncer-j7kp4\" (UID: \"eceae17b-5b0b-45ce-87cf-9d720583c247\") " pod="kube-system/global-pull-secret-syncer-j7kp4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981460 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-node-log\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981462 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-cni-dir\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981467 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-etc-systemd\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981485 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207974a1-670c-49f2-b0fb-3946d261e608-system-cni-dir\") pod \"multus-additional-cni-plugins-pq2jl\" (UID: \"207974a1-670c-49f2-b0fb-3946d261e608\") " pod="openshift-multus/multus-additional-cni-plugins-pq2jl" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981487 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981530 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-socket-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981498 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-node-log\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981532 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981557 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-netns\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981585 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-run\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981595 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77546eb8-8ef9-46b1-a47a-ca80ca5f76ee-host-run-netns\") pod \"ovnkube-node-8sfkw\" (UID: \"77546eb8-8ef9-46b1-a47a-ca80ca5f76ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981605 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-socket-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981613 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981641 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-socket-dir-parent\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981649 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e413f92-3561-4f46-a45b-1fbe4919af71-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6d9js\" (UID: \"5e413f92-3561-4f46-a45b-1fbe4919af71\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6d9js" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981654 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/caf2ae77-8244-44fa-a47f-1b8555fed1f2-run\") pod \"tuned-vj7jb\" (UID: \"caf2ae77-8244-44fa-a47f-1b8555fed1f2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj7jb" Nov 02 19:48:22.981655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981610 5800 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 19:48:22.983868 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981720 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7bd90b6c-66f9-4ea8-8f73-2981acce43db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:22.983868 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.981721 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21-multus-socket-dir-parent\") pod \"multus-5slm4\" (UID: \"bdbc31d0-a7a3-4898-8841-9ed8ebf5ff21\") " pod="openshift-multus/multus-5slm4" Nov 02 19:48:22.983868 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:22.983738 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7bd90b6c-66f9-4ea8-8f73-2981acce43db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6hpx\" (UID: \"7bd90b6c-66f9-4ea8-8f73-2981acce43db\") " pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:23.014881 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.014853 5800 scope.go:122] "RemoveContainer" containerID="af3dc24e1e3b99eeaabf84378cdedff256741f25d6a0946fadd6d653d68e71c2" Nov 02 19:48:23.015933 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.015903 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:23.029237 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.029213 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6hpx" Nov 02 19:48:23.493184 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.493068 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x6hpx"] Nov 02 19:48:23.497091 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:23.497045 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd90b6c_66f9_4ea8_8f73_2981acce43db.slice/crio-629de60e3a17baf92531b0165ff665ae1f5c553a80efdcb28692547066f34514 WatchSource:0}: Error finding container 629de60e3a17baf92531b0165ff665ae1f5c553a80efdcb28692547066f34514: Status 404 returned error can't find the container with id 629de60e3a17baf92531b0165ff665ae1f5c553a80efdcb28692547066f34514 Nov 02 19:48:23.594527 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.594503 5800 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:48:23.680726 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.680698 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:23.690397 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.690355 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6hpx" event={"ID":"7bd90b6c-66f9-4ea8-8f73-2981acce43db","Type":"ContainerStarted","Data":"b176218e5cbd0a1a728d1b67356303e3c15c11d93d3c6a280206334c6b47112f"} Nov 02 19:48:23.690537 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.690413 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6hpx" event={"ID":"7bd90b6c-66f9-4ea8-8f73-2981acce43db","Type":"ContainerStarted","Data":"629de60e3a17baf92531b0165ff665ae1f5c553a80efdcb28692547066f34514"} Nov 02 19:48:23.692035 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.692015 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:48:23.692172 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.692152 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" event={"ID":"deaf3243-1eeb-4d94-bef9-524b67035ba2","Type":"ContainerStarted","Data":"0854c9b7f4c91ab2311e50137a100722d5c3779eb9e8acdea46aca7c91895281"} Nov 02 19:48:23.743586 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:23.743503 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:24.017178 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:24.017152 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:24.608501 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:24.604468 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bffd989f9-r4bss"] Nov 02 19:48:24.697331 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:24.697303 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f6668f8f7-2vjcr" Nov 02 19:48:25.588122 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.588042 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-27pq5" Nov 02 19:48:25.701532 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.701498 5800 generic.go:356] "Generic (PLEG): container finished" podID="0fb381a6-0d14-47a9-a144-4fd8be651947" containerID="f440c72623dba03fdb7a5ef29a0f045f77797c0557980910c67e6acf3391aab8" exitCode=0 Nov 02 19:48:25.701969 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.701571 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kg2ng" event={"ID":"0fb381a6-0d14-47a9-a144-4fd8be651947","Type":"ContainerDied","Data":"f440c72623dba03fdb7a5ef29a0f045f77797c0557980910c67e6acf3391aab8"} Nov 02 19:48:25.702040 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.701981 5800 scope.go:122] "RemoveContainer" containerID="f440c72623dba03fdb7a5ef29a0f045f77797c0557980910c67e6acf3391aab8" Nov 02 19:48:25.703650 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.703628 5800 generic.go:356] "Generic (PLEG): container finished" podID="b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f" containerID="df497ecc58c812385314dfab5c4d2bc33e784ea26d82899100156df015df60b2" exitCode=0 Nov 02 19:48:25.703748 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.703700 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" event={"ID":"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f","Type":"ContainerDied","Data":"df497ecc58c812385314dfab5c4d2bc33e784ea26d82899100156df015df60b2"} Nov 02 19:48:25.703997 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.703978 5800 scope.go:122] "RemoveContainer" containerID="df497ecc58c812385314dfab5c4d2bc33e784ea26d82899100156df015df60b2" Nov 02 19:48:25.705733 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.705713 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6hpx" event={"ID":"7bd90b6c-66f9-4ea8-8f73-2981acce43db","Type":"ContainerStarted","Data":"ea1f0d5f2136b3e65c3f82fe48072a1d27aa486792b7a7dbd61b6760e2826f9b"} Nov 02 19:48:25.707173 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.707152 5800 generic.go:356] "Generic (PLEG): container finished" podID="ff626920-a094-4782-aae1-ac6bc9d0b359" containerID="6e3d7e340191b99edfd9108e3b57f2c6705e1de039b067c15db8d212cf812b3c" exitCode=0 Nov 02 19:48:25.707268 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.707221 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" event={"ID":"ff626920-a094-4782-aae1-ac6bc9d0b359","Type":"ContainerDied","Data":"6e3d7e340191b99edfd9108e3b57f2c6705e1de039b067c15db8d212cf812b3c"} Nov 02 19:48:25.707688 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.707673 5800 scope.go:122] "RemoveContainer" containerID="6e3d7e340191b99edfd9108e3b57f2c6705e1de039b067c15db8d212cf812b3c" Nov 02 19:48:25.910545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.910448 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:25.915561 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.915422 5800 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:48:25.996964 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:25.996933 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:26.230156 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:26.230075 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8sfkw" Nov 02 19:48:26.713480 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:26.713439 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-lv8xc" event={"ID":"ff626920-a094-4782-aae1-ac6bc9d0b359","Type":"ContainerStarted","Data":"1b4feb4a28a64b7bcf03d462eed76110175ceaad4fd31216403d69fe594ca559"} Nov 02 19:48:26.715883 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:26.715848 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kg2ng" event={"ID":"0fb381a6-0d14-47a9-a144-4fd8be651947","Type":"ContainerStarted","Data":"81a55c736d5ea9fc4c963234b8b40e4ab42c5723e595021e1e9b2be95e8b53e9"} Nov 02 19:48:26.717722 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:26.717693 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-5x8qv" event={"ID":"b0b42a16-a3fd-4eeb-8fbe-cb1b5a40fd6f","Type":"ContainerStarted","Data":"ac6565af06d6aaed5e3ba27ffbb5b53352732f5600590490da78850701325969"} Nov 02 19:48:27.723888 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:27.723799 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6hpx" event={"ID":"7bd90b6c-66f9-4ea8-8f73-2981acce43db","Type":"ContainerStarted","Data":"94ba46dc1b24bb4498efd420e80e9964fea6054d0e4795348b5b9aac3ed03a78"} Nov 02 19:48:28.741770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:28.741740 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:28.745591 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:28.745567 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d458f586f-ndk5f" Nov 02 19:48:31.213602 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.213522 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:31.213996 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.213662 5800 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 19:48:31.217563 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.217536 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-2b245" Nov 02 19:48:31.409465 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.409428 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9xf99"] Nov 02 19:48:31.413087 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.413066 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453530 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453497 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-textfile\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453530 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453531 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453773 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453569 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39bb832f-2158-430f-996c-b715e3c262c1-metrics-client-ca\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453773 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453669 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-tls\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453773 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453693 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-wtmp\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453773 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453709 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-root\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453773 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453727 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.453773 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453746 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-sys\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.454093 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.453836 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcwt\" (UniqueName: \"kubernetes.io/projected/39bb832f-2158-430f-996c-b715e3c262c1-kube-api-access-4wcwt\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.465745 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.465674 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 19:48:31.465745 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.465736 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 19:48:31.532727 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.532674 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 19:48:31.532889 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.532762 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 19:48:31.532889 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.532772 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v8hmw\"" Nov 02 19:48:31.554567 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554542 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-textfile\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554708 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554569 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554708 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554604 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39bb832f-2158-430f-996c-b715e3c262c1-metrics-client-ca\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554708 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554630 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-tls\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554708 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554661 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-wtmp\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554708 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554683 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-root\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554958 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554709 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554958 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554732 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-sys\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554958 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554766 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcwt\" (UniqueName: \"kubernetes.io/projected/39bb832f-2158-430f-996c-b715e3c262c1-kube-api-access-4wcwt\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554958 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554768 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-root\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554958 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554851 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-wtmp\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.554958 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.554861 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39bb832f-2158-430f-996c-b715e3c262c1-sys\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.555194 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.555016 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-textfile\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.555365 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.555347 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39bb832f-2158-430f-996c-b715e3c262c1-metrics-client-ca\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.555438 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.555397 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.556997 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.556976 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.557114 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.557095 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39bb832f-2158-430f-996c-b715e3c262c1-node-exporter-tls\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.607537 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.607507 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcwt\" (UniqueName: \"kubernetes.io/projected/39bb832f-2158-430f-996c-b715e3c262c1-kube-api-access-4wcwt\") pod \"node-exporter-9xf99\" (UID: \"39bb832f-2158-430f-996c-b715e3c262c1\") " pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.729330 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:31.729232 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9xf99" Nov 02 19:48:31.745967 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:48:31.745934 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bb832f_2158_430f_996c_b715e3c262c1.slice/crio-3d9b2f36330e93063c028212addc350f6be4b25a593835d4123f002f953863e2 WatchSource:0}: Error finding container 3d9b2f36330e93063c028212addc350f6be4b25a593835d4123f002f953863e2: Status 404 returned error can't find the container with id 3d9b2f36330e93063c028212addc350f6be4b25a593835d4123f002f953863e2 Nov 02 19:48:32.206199 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:32.206159 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:32.210275 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:32.210253 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:32.761214 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:32.761180 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xf99" event={"ID":"39bb832f-2158-430f-996c-b715e3c262c1","Type":"ContainerStarted","Data":"3d9b2f36330e93063c028212addc350f6be4b25a593835d4123f002f953863e2"} Nov 02 19:48:33.771097 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:33.771059 5800 generic.go:356] "Generic (PLEG): container finished" podID="39bb832f-2158-430f-996c-b715e3c262c1" containerID="8bd25e983c8318589104289ff4b3387769efa2efff9a7f211cd777aaa40c629e" exitCode=0 Nov 02 19:48:33.771503 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:33.771111 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xf99" event={"ID":"39bb832f-2158-430f-996c-b715e3c262c1","Type":"ContainerDied","Data":"8bd25e983c8318589104289ff4b3387769efa2efff9a7f211cd777aaa40c629e"} Nov 02 19:48:34.777204 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:34.777166 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xf99" event={"ID":"39bb832f-2158-430f-996c-b715e3c262c1","Type":"ContainerStarted","Data":"07239b43ad22f4019e60d52e9a30632a69ca97b1270035cfe253814b91355d31"} Nov 02 19:48:34.777598 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:34.777233 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9xf99" event={"ID":"39bb832f-2158-430f-996c-b715e3c262c1","Type":"ContainerStarted","Data":"6b5a68d8d30338a162625d20969931d74d54bb16633eeef8fd6ea2e802bdfcd5"} Nov 02 19:48:34.933118 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:34.933060 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9xf99" podStartSLOduration=2.139767863 podStartE2EDuration="3.933043453s" totalImagesPullingTime="1.725946899s" totalInitContainerRuntime="67.328691ms" isStatefulPod=true podCreationTimestamp="2026-11-02 19:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:48:34.932237347 +0000 UTC m=+13.449009932" watchObservedRunningTime="2026-11-02 19:48:34.933043453 +0000 UTC m=+13.449816059" Nov 02 19:48:39.980679 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:39.980622 5800 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-trg79_7c9766ef-0c1a-4ddc-9918-e0c1fb68aada/konnectivity-agent/0.log" limit=1024 Nov 02 19:48:42.164707 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:42.164677 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-trg79" Nov 02 19:48:44.451606 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.451566 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gcdjm-447t7"] Nov 02 19:48:44.452018 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.451761 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-gcdjm-447t7" podUID="674789e3-135f-4296-9de6-274e37588b04" containerName="insights-gathering" containerID="cri-o://c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f" gracePeriod=30 Nov 02 19:48:44.713477 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.713415 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:44.775647 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.775616 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\" (UniqueName: \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\") pod \"674789e3-135f-4296-9de6-274e37588b04\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " Nov 02 19:48:44.775797 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.775669 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\") pod \"674789e3-135f-4296-9de6-274e37588b04\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " Nov 02 19:48:44.775797 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.775733 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\") pod \"674789e3-135f-4296-9de6-274e37588b04\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " Nov 02 19:48:44.775874 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.775837 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\") pod \"674789e3-135f-4296-9de6-274e37588b04\" (UID: \"674789e3-135f-4296-9de6-274e37588b04\") " Nov 02 19:48:44.776140 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.776107 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle" pod "674789e3-135f-4296-9de6-274e37588b04" (UID: "674789e3-135f-4296-9de6-274e37588b04"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:44.776260 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.776135 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle" pod "674789e3-135f-4296-9de6-274e37588b04" (UID: "674789e3-135f-4296-9de6-274e37588b04"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:44.776260 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.776154 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path" pod "674789e3-135f-4296-9de6-274e37588b04" (UID: "674789e3-135f-4296-9de6-274e37588b04"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:48:44.777933 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.777914 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c" pod "674789e3-135f-4296-9de6-274e37588b04" (UID: "674789e3-135f-4296-9de6-274e37588b04"). InnerVolumeSpecName "kube-api-access-5bh5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:44.815778 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.815743 5800 generic.go:356] "Generic (PLEG): container finished" podID="674789e3-135f-4296-9de6-274e37588b04" containerID="c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f" exitCode=2 Nov 02 19:48:44.815921 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.815788 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gcdjm-447t7" Nov 02 19:48:44.815921 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.815833 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcdjm-447t7" event={"ID":"674789e3-135f-4296-9de6-274e37588b04","Type":"ContainerDied","Data":"c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f"} Nov 02 19:48:44.815921 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.815883 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gcdjm-447t7" event={"ID":"674789e3-135f-4296-9de6-274e37588b04","Type":"ContainerDied","Data":"28e75b5fd0566e209cf8ea9fbb570f6177c3cb18782286d48f9b992b46ee3990"} Nov 02 19:48:44.815921 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.815904 5800 scope.go:122] "RemoveContainer" containerID="c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f" Nov 02 19:48:44.825670 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.825650 5800 scope.go:122] "RemoveContainer" containerID="c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f" Nov 02 19:48:44.826010 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:44.825991 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f\": container with ID starting with c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f not found: ID does not exist" containerID="c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f" Nov 02 19:48:44.826075 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.826021 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f"} err="failed to get container status \"c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f\": rpc error: code = NotFound desc = could not find container \"c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f\": container with ID starting with c382fbe06b0a6c4ac96622dc0aeffe3b405fd19c03ab53e013c34466541ad26f not found: ID does not exist" Nov 02 19:48:44.876639 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.876596 5800 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/674789e3-135f-4296-9de6-274e37588b04-archives-path\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:44.876639 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.876635 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bh5c\" (UniqueName: \"kubernetes.io/projected/674789e3-135f-4296-9de6-274e37588b04-kube-api-access-5bh5c\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:44.876639 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.876649 5800 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-service-ca-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:44.876862 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.876662 5800 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/674789e3-135f-4296-9de6-274e37588b04-trusted-ca-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:44.972044 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.971974 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gcdjm-447t7"] Nov 02 19:48:44.975577 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:44.975553 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-gcdjm-447t7"] Nov 02 19:48:45.619461 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:45.619428 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674789e3-135f-4296-9de6-274e37588b04" path="/var/lib/kubelet/pods/674789e3-135f-4296-9de6-274e37588b04/volumes" Nov 02 19:48:45.999746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:45.999716 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mpxzs" Nov 02 19:48:49.631477 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:49.631440 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" podUID="fe5633bb-4256-4685-943e-0e0ecbce30cd" containerName="registry" containerID="cri-o://6c79c2ff2528f0aa8829f2452b1a4fd8d52c06218e22fccbf44a019ba129d502" gracePeriod=30 Nov 02 19:48:49.838154 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:49.838116 5800 generic.go:356] "Generic (PLEG): container finished" podID="fe5633bb-4256-4685-943e-0e0ecbce30cd" containerID="6c79c2ff2528f0aa8829f2452b1a4fd8d52c06218e22fccbf44a019ba129d502" exitCode=0 Nov 02 19:48:49.838322 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:49.838191 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" event={"ID":"fe5633bb-4256-4685-943e-0e0ecbce30cd","Type":"ContainerDied","Data":"6c79c2ff2528f0aa8829f2452b1a4fd8d52c06218e22fccbf44a019ba129d502"} Nov 02 19:48:49.881027 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:49.881003 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:50.024328 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024294 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.024523 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024341 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.024523 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024371 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.024523 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024397 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.024523 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024482 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.024523 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024516 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.024776 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024577 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.024776 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.024615 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\") pod \"fe5633bb-4256-4685-943e-0e0ecbce30cd\" (UID: \"fe5633bb-4256-4685-943e-0e0ecbce30cd\") " Nov 02 19:48:50.025084 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.025028 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:50.025215 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.025131 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:48:50.027056 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.027023 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "kube-api-access-jckjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:50.027195 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.027099 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:50.027195 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.027142 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:50.027195 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.027181 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:48:50.027389 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.027309 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:48:50.031714 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.031651 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted" pod "fe5633bb-4256-4685-943e-0e0ecbce30cd" (UID: "fe5633bb-4256-4685-943e-0e0ecbce30cd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:48:50.125701 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125663 5800 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-trusted-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.125701 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125694 5800 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-tls\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.125701 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125707 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jckjq\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-kube-api-access-jckjq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.125941 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125720 5800 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe5633bb-4256-4685-943e-0e0ecbce30cd-bound-sa-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.125941 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125734 5800 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-image-registry-private-configuration\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.125941 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125746 5800 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe5633bb-4256-4685-943e-0e0ecbce30cd-ca-trust-extracted\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.125941 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125759 5800 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe5633bb-4256-4685-943e-0e0ecbce30cd-installation-pull-secrets\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.125941 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.125771 5800 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe5633bb-4256-4685-943e-0e0ecbce30cd-registry-certificates\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:48:50.843096 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.843058 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" event={"ID":"fe5633bb-4256-4685-943e-0e0ecbce30cd","Type":"ContainerDied","Data":"fb3c12577c50a9086bff5cb610c3c0106ca8c472ac4f542126120a94c382763b"} Nov 02 19:48:50.843583 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.843120 5800 scope.go:122] "RemoveContainer" containerID="6c79c2ff2528f0aa8829f2452b1a4fd8d52c06218e22fccbf44a019ba129d502" Nov 02 19:48:50.843583 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:50.843068 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bffd989f9-r4bss" Nov 02 19:48:51.042682 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:51.042650 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bffd989f9-r4bss"] Nov 02 19:48:51.089144 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:51.089106 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bffd989f9-r4bss"] Nov 02 19:48:51.619500 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:48:51.619466 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5633bb-4256-4685-943e-0e0ecbce30cd" path="/var/lib/kubelet/pods/fe5633bb-4256-4685-943e-0e0ecbce30cd/volumes" Nov 02 19:48:52.694490 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:48:52.694451 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:49:22.694740 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:49:22.694695 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:49:52.695030 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:49:52.694989 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:50:22.693442 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:50:22.693356 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:50:52.694481 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:50:52.694421 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:51:22.695224 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:51:22.695184 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:51:52.695697 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:51:52.695660 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:52:22.694705 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:52:22.694657 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:52:29.468784 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.468745 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh"] Nov 02 19:52:29.469316 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.469191 5800 cpu_manager.go:414] "containerMap: removing container" podUID="fe5633bb-4256-4685-943e-0e0ecbce30cd" containerName="registry" Nov 02 19:52:29.469316 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.469209 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5633bb-4256-4685-943e-0e0ecbce30cd" containerName="registry" Nov 02 19:52:29.469316 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.469243 5800 cpu_manager.go:414] "containerMap: removing container" podUID="674789e3-135f-4296-9de6-274e37588b04" containerName="insights-gathering" Nov 02 19:52:29.469316 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.469250 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674789e3-135f-4296-9de6-274e37588b04" containerName="insights-gathering" Nov 02 19:52:29.469537 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.469355 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5633bb-4256-4685-943e-0e0ecbce30cd" containerName="registry" Nov 02 19:52:29.469537 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.469373 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="674789e3-135f-4296-9de6-274e37588b04" containerName="insights-gathering" Nov 02 19:52:29.472639 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.472615 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.558928 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.558893 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.558928 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.558931 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.559134 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.558969 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7v6\" (UniqueName: \"kubernetes.io/projected/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-kube-api-access-kp7v6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.570959 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.570936 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 19:52:29.571138 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.571120 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 19:52:29.571301 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.571261 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-2wkg4\"" Nov 02 19:52:29.571987 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.571968 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh"] Nov 02 19:52:29.659538 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.659507 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.659728 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.659545 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.659728 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.659601 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7v6\" (UniqueName: \"kubernetes.io/projected/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-kube-api-access-kp7v6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.659899 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.659878 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.659957 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.659912 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.712688 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.712661 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7v6\" (UniqueName: \"kubernetes.io/projected/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-kube-api-access-kp7v6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:29.788819 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:29.788790 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:52:30.148625 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:30.148546 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh"] Nov 02 19:52:30.151500 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:52:30.151470 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7c6a17_2912_4abd_90a7_7ef1ff4901b3.slice/crio-5d81db5d37e45779c8b882d7d441054383768f060d37d13f1a6851fa61e9ea69 WatchSource:0}: Error finding container 5d81db5d37e45779c8b882d7d441054383768f060d37d13f1a6851fa61e9ea69: Status 404 returned error can't find the container with id 5d81db5d37e45779c8b882d7d441054383768f060d37d13f1a6851fa61e9ea69 Nov 02 19:52:30.641015 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:30.640977 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" event={"ID":"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3","Type":"ContainerStarted","Data":"5d81db5d37e45779c8b882d7d441054383768f060d37d13f1a6851fa61e9ea69"} Nov 02 19:52:31.981796 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:31.981763 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279"] Nov 02 19:52:31.985295 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:31.985265 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.082050 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.082015 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.082213 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.082073 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcthb\" (UniqueName: \"kubernetes.io/projected/60bdb00e-3492-46b5-83bd-4189210aa89d-kube-api-access-lcthb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.082213 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.082172 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.083174 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.083157 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rn4dc\"" Nov 02 19:52:32.089240 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.089197 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279"] Nov 02 19:52:32.183323 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.183262 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.183514 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.183374 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.183514 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.183428 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcthb\" (UniqueName: \"kubernetes.io/projected/60bdb00e-3492-46b5-83bd-4189210aa89d-kube-api-access-lcthb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.183686 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.183661 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.183747 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.183693 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.236730 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.236657 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcthb\" (UniqueName: \"kubernetes.io/projected/60bdb00e-3492-46b5-83bd-4189210aa89d-kube-api-access-lcthb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.301768 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.301731 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:52:32.674146 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:32.674114 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279"] Nov 02 19:52:32.711174 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:52:32.711138 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bdb00e_3492_46b5_83bd_4189210aa89d.slice/crio-615c202fdfa8e28ce24e3801878a97e24564f336cca0087fd498bfd10135fed2 WatchSource:0}: Error finding container 615c202fdfa8e28ce24e3801878a97e24564f336cca0087fd498bfd10135fed2: Status 404 returned error can't find the container with id 615c202fdfa8e28ce24e3801878a97e24564f336cca0087fd498bfd10135fed2 Nov 02 19:52:33.654617 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:33.654577 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" event={"ID":"60bdb00e-3492-46b5-83bd-4189210aa89d","Type":"ContainerStarted","Data":"615c202fdfa8e28ce24e3801878a97e24564f336cca0087fd498bfd10135fed2"} Nov 02 19:52:34.285795 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.285757 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6"] Nov 02 19:52:34.289305 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.289269 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.338484 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.338452 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 19:52:34.369432 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.369401 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\"" Nov 02 19:52:34.386098 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.386066 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6"] Nov 02 19:52:34.403464 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403431 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwq9l\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-kube-api-access-qwq9l\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403612 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403481 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403612 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403548 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403731 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403676 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403731 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403712 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403833 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403745 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403886 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403839 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403943 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403882 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.403943 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.403907 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.445878 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.445842 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz"] Nov 02 19:52:34.449760 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.449733 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.474887 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.474854 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk"] Nov 02 19:52:34.478703 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.478681 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.495855 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.495830 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx"] Nov 02 19:52:34.498923 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.498899 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9vwqk\"" Nov 02 19:52:34.499456 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.499434 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.503789 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.503733 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz"] Nov 02 19:52:34.505455 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505415 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505455 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505451 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505680 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505509 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwq9l\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-kube-api-access-qwq9l\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505680 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505538 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505680 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505597 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505680 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505625 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505680 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505666 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505997 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505710 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.505997 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.505740 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.506170 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.506102 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.506505 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.506458 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.506712 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.506691 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.506856 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.506823 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.507111 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.507089 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.509803 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.509768 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.509900 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.509833 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.530500 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.530477 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d"] Nov 02 19:52:34.534489 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.534170 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk"] Nov 02 19:52:34.534489 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.534301 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.540358 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.540305 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-dxlc4\"" Nov 02 19:52:34.547844 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.547822 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-2hf7r\"" Nov 02 19:52:34.556946 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.556919 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx"] Nov 02 19:52:34.560184 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.560158 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.560364 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.560343 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwq9l\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-kube-api-access-qwq9l\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-slhh6\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.590463 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.590412 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d"] Nov 02 19:52:34.598925 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.598901 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ctw8r\"" Nov 02 19:52:34.602139 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.602112 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:34.607017 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.606992 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.607129 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607044 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.607129 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607070 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.607129 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607098 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9dvt\" (UniqueName: \"kubernetes.io/projected/49613984-cc5e-4db0-ba7d-2a6d321f7033-kube-api-access-c9dvt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.607129 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607122 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.607355 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607165 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.607355 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607220 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874h6\" (UniqueName: \"kubernetes.io/projected/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-kube-api-access-874h6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.607355 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607251 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkxt\" (UniqueName: \"kubernetes.io/projected/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-kube-api-access-6tkxt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.607616 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.607552 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.708143 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708099 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708185 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708212 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708243 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dvt\" (UniqueName: \"kubernetes.io/projected/49613984-cc5e-4db0-ba7d-2a6d321f7033-kube-api-access-c9dvt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708269 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708337 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708370 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708424 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874h6\" (UniqueName: \"kubernetes.io/projected/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-kube-api-access-874h6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708454 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkxt\" (UniqueName: \"kubernetes.io/projected/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-kube-api-access-6tkxt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708495 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708521 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4z54\" (UniqueName: \"kubernetes.io/projected/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-kube-api-access-l4z54\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708564 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.708607 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708586 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.709193 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708610 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.709193 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708621 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.709193 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708864 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.709193 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.708921 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.709340 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.709268 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.765221 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.765192 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkxt\" (UniqueName: \"kubernetes.io/projected/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-kube-api-access-6tkxt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.765418 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.765354 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874h6\" (UniqueName: \"kubernetes.io/projected/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-kube-api-access-874h6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.765418 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.765354 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9dvt\" (UniqueName: \"kubernetes.io/projected/49613984-cc5e-4db0-ba7d-2a6d321f7033-kube-api-access-c9dvt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.769935 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.769909 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:52:34.797628 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.797557 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:52:34.809802 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.809773 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.809899 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.809857 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.809958 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.809886 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4z54\" (UniqueName: \"kubernetes.io/projected/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-kube-api-access-l4z54\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.810214 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.810192 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.810450 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.810430 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:34.825888 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.825756 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:52:34.865759 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:34.865675 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4z54\" (UniqueName: \"kubernetes.io/projected/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-kube-api-access-l4z54\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:35.036208 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.036183 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6"] Nov 02 19:52:35.038330 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:52:35.038272 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfdfd09_0256_4ca4_a4d1_c99ec864c822.slice/crio-cfb0d29ef68e5b965f76cabccfadb57e3364daaecc7671228ef3c1a5cb74c96b WatchSource:0}: Error finding container cfb0d29ef68e5b965f76cabccfadb57e3364daaecc7671228ef3c1a5cb74c96b: Status 404 returned error can't find the container with id cfb0d29ef68e5b965f76cabccfadb57e3364daaecc7671228ef3c1a5cb74c96b Nov 02 19:52:35.143496 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.143469 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz"] Nov 02 19:52:35.146092 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:52:35.146052 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faa1e1a_fcb8_421f_8cc6_744f6691c98b.slice/crio-c2c0b67c6aaf124859678fc2d5c200d35141e517e287d3f70e78b22bcc9a2ea9 WatchSource:0}: Error finding container c2c0b67c6aaf124859678fc2d5c200d35141e517e287d3f70e78b22bcc9a2ea9: Status 404 returned error can't find the container with id c2c0b67c6aaf124859678fc2d5c200d35141e517e287d3f70e78b22bcc9a2ea9 Nov 02 19:52:35.154817 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.154794 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:52:35.196072 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.194319 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk"] Nov 02 19:52:35.199484 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:52:35.198409 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49613984_cc5e_4db0_ba7d_2a6d321f7033.slice/crio-bc8ac98b1c484bf4a50d9cffbb509166396d8670f39d1edf63d351862764b34b WatchSource:0}: Error finding container bc8ac98b1c484bf4a50d9cffbb509166396d8670f39d1edf63d351862764b34b: Status 404 returned error can't find the container with id bc8ac98b1c484bf4a50d9cffbb509166396d8670f39d1edf63d351862764b34b Nov 02 19:52:35.218402 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.217941 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx"] Nov 02 19:52:35.222670 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:52:35.222647 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d74a24_b691_4d94_bf5f_7ad1f0c662f7.slice/crio-02d572cbbfd3a7502cc3d84c369204bc7e5c86022b489c02e68d4186a272bc7e WatchSource:0}: Error finding container 02d572cbbfd3a7502cc3d84c369204bc7e5c86022b489c02e68d4186a272bc7e: Status 404 returned error can't find the container with id 02d572cbbfd3a7502cc3d84c369204bc7e5c86022b489c02e68d4186a272bc7e Nov 02 19:52:35.529483 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.529412 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d"] Nov 02 19:52:35.533892 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:52:35.533843 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb146c37_4fbd_46f1_8bfb_b9eb3c88bba1.slice/crio-dd8e5888305f6c11697d26d6194abb9a51a52517d4b023abedf39516ad5f1354 WatchSource:0}: Error finding container dd8e5888305f6c11697d26d6194abb9a51a52517d4b023abedf39516ad5f1354: Status 404 returned error can't find the container with id dd8e5888305f6c11697d26d6194abb9a51a52517d4b023abedf39516ad5f1354 Nov 02 19:52:35.678935 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.678896 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" event={"ID":"49613984-cc5e-4db0-ba7d-2a6d321f7033","Type":"ContainerStarted","Data":"bc8ac98b1c484bf4a50d9cffbb509166396d8670f39d1edf63d351862764b34b"} Nov 02 19:52:35.680382 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.680347 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" event={"ID":"5dfdfd09-0256-4ca4-a4d1-c99ec864c822","Type":"ContainerStarted","Data":"cfb0d29ef68e5b965f76cabccfadb57e3364daaecc7671228ef3c1a5cb74c96b"} Nov 02 19:52:35.681818 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.681791 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" event={"ID":"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1","Type":"ContainerStarted","Data":"dd8e5888305f6c11697d26d6194abb9a51a52517d4b023abedf39516ad5f1354"} Nov 02 19:52:35.683225 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.683196 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" event={"ID":"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7","Type":"ContainerStarted","Data":"02d572cbbfd3a7502cc3d84c369204bc7e5c86022b489c02e68d4186a272bc7e"} Nov 02 19:52:35.684979 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:35.684953 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" event={"ID":"1faa1e1a-fcb8-421f-8cc6-744f6691c98b","Type":"ContainerStarted","Data":"c2c0b67c6aaf124859678fc2d5c200d35141e517e287d3f70e78b22bcc9a2ea9"} Nov 02 19:52:42.479379 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:42.479334 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:42.479823 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:42.479420 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:42.479823 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:42.479467 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:52:42.739253 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:42.739156 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" event={"ID":"5dfdfd09-0256-4ca4-a4d1-c99ec864c822","Type":"ContainerStarted","Data":"934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004"} Nov 02 19:52:42.942567 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:42.942503 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podStartSLOduration=1.5048725040000002 podStartE2EDuration="8.942483707s" totalImagesPullingTime="7.437611203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:52:42.895788237 +0000 UTC m=+261.412560835" watchObservedRunningTime="2026-11-02 19:52:42.942483707 +0000 UTC m=+261.459256295" Nov 02 19:52:43.604794 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:43.604668 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:43.612331 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:43.610579 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:43.745706 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:43.745291 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:43.746502 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:43.746431 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:52:44.754198 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:44.754160 5800 generic.go:356] "Generic (PLEG): container finished" podID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerID="6374295f8ee707e4bf72617afb70233b2942c0bbb7d80e9c09d5ecfe4f374ce1" exitCode=0 Nov 02 19:52:44.755926 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:44.755877 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" event={"ID":"60bdb00e-3492-46b5-83bd-4189210aa89d","Type":"ContainerDied","Data":"6374295f8ee707e4bf72617afb70233b2942c0bbb7d80e9c09d5ecfe4f374ce1"} Nov 02 19:52:45.763299 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:45.763113 5800 generic.go:356] "Generic (PLEG): container finished" podID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerID="e4a1af4b0c005c17accf675aefb6b13b46fcb43d5948c843bc2005b6d57095fa" exitCode=0 Nov 02 19:52:45.763299 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:45.763201 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" event={"ID":"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3","Type":"ContainerDied","Data":"e4a1af4b0c005c17accf675aefb6b13b46fcb43d5948c843bc2005b6d57095fa"} Nov 02 19:52:45.766327 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:45.766300 5800 generic.go:356] "Generic (PLEG): container finished" podID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerID="552efb3b3c63a34e8db2565e1f7296ee54d62232a8c749218e0ecf3ed0249c0f" exitCode=0 Nov 02 19:52:45.767010 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:45.766977 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" event={"ID":"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1","Type":"ContainerDied","Data":"552efb3b3c63a34e8db2565e1f7296ee54d62232a8c749218e0ecf3ed0249c0f"} Nov 02 19:52:47.778139 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:47.778074 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" event={"ID":"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7","Type":"ContainerStarted","Data":"72df87e6fd2bfbef7e28969de142b31ae7f14c313553612a3f8313ba299d6cf7"} Nov 02 19:52:47.779925 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:47.779893 5800 generic.go:356] "Generic (PLEG): container finished" podID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerID="ad47fc84ba62dc53713c167b9732738a31de420d2a1b62de07999bd56c9b68b9" exitCode=0 Nov 02 19:52:47.780017 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:47.779950 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" event={"ID":"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3","Type":"ContainerDied","Data":"ad47fc84ba62dc53713c167b9732738a31de420d2a1b62de07999bd56c9b68b9"} Nov 02 19:52:48.786707 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.786664 5800 generic.go:356] "Generic (PLEG): container finished" podID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerID="f0bb1b6a7d8f2c4f7e239b23f9260eb92278cee702cbafdb5f5ba333da31f1e5" exitCode=0 Nov 02 19:52:48.786707 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.786692 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" event={"ID":"49613984-cc5e-4db0-ba7d-2a6d321f7033","Type":"ContainerDied","Data":"f0bb1b6a7d8f2c4f7e239b23f9260eb92278cee702cbafdb5f5ba333da31f1e5"} Nov 02 19:52:48.790158 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.790074 5800 generic.go:356] "Generic (PLEG): container finished" podID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerID="518884ac6eefe98032569336d1442d039d2f319714d409d4011c7f5aa434c87a" exitCode=0 Nov 02 19:52:48.790158 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.790109 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" event={"ID":"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1","Type":"ContainerDied","Data":"518884ac6eefe98032569336d1442d039d2f319714d409d4011c7f5aa434c87a"} Nov 02 19:52:48.791886 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.791860 5800 generic.go:356] "Generic (PLEG): container finished" podID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerID="72df87e6fd2bfbef7e28969de142b31ae7f14c313553612a3f8313ba299d6cf7" exitCode=0 Nov 02 19:52:48.791982 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.791949 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" event={"ID":"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7","Type":"ContainerDied","Data":"72df87e6fd2bfbef7e28969de142b31ae7f14c313553612a3f8313ba299d6cf7"} Nov 02 19:52:48.793713 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.793649 5800 generic.go:356] "Generic (PLEG): container finished" podID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerID="a1fba61258ca5694c8bf4ba7812a4c28a96b4dfcac368457dae085bb1100c741" exitCode=0 Nov 02 19:52:48.793713 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:48.793707 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" event={"ID":"1faa1e1a-fcb8-421f-8cc6-744f6691c98b","Type":"ContainerDied","Data":"a1fba61258ca5694c8bf4ba7812a4c28a96b4dfcac368457dae085bb1100c741"} Nov 02 19:52:49.800847 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:49.800802 5800 generic.go:356] "Generic (PLEG): container finished" podID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerID="ca4fa0c59c78bc7a107998a4e13e3d8933e7b40b5906db96b3cebe4fd007fc2c" exitCode=0 Nov 02 19:52:49.801300 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:49.800901 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" event={"ID":"60bdb00e-3492-46b5-83bd-4189210aa89d","Type":"ContainerDied","Data":"ca4fa0c59c78bc7a107998a4e13e3d8933e7b40b5906db96b3cebe4fd007fc2c"} Nov 02 19:52:50.808312 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:50.808263 5800 generic.go:356] "Generic (PLEG): container finished" podID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerID="bd602485b602e3f0107e937a4233e99aa5c52ded6d5edcf86855cf939c54bcfe" exitCode=0 Nov 02 19:52:50.808678 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:50.808416 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" event={"ID":"1faa1e1a-fcb8-421f-8cc6-744f6691c98b","Type":"ContainerDied","Data":"bd602485b602e3f0107e937a4233e99aa5c52ded6d5edcf86855cf939c54bcfe"} Nov 02 19:52:50.811969 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:50.811945 5800 generic.go:356] "Generic (PLEG): container finished" podID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerID="48df6fb3cbfc5341ac0ab6635deac928065e3465759ff48c00b716ed92fceaee" exitCode=0 Nov 02 19:52:50.812082 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:50.812065 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" event={"ID":"49613984-cc5e-4db0-ba7d-2a6d321f7033","Type":"ContainerDied","Data":"48df6fb3cbfc5341ac0ab6635deac928065e3465759ff48c00b716ed92fceaee"} Nov 02 19:52:50.814973 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:50.814949 5800 generic.go:356] "Generic (PLEG): container finished" podID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerID="945a34ef74cff7b5e82b638d7998c29f9823cd2482ca8e67e638f99c67b4d1b1" exitCode=0 Nov 02 19:52:50.815030 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:50.814985 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" event={"ID":"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7","Type":"ContainerDied","Data":"945a34ef74cff7b5e82b638d7998c29f9823cd2482ca8e67e638f99c67b4d1b1"} Nov 02 19:52:52.743566 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:52:52.743521 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:52:59.883980 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:59.883937 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" event={"ID":"1faa1e1a-fcb8-421f-8cc6-744f6691c98b","Type":"ContainerStarted","Data":"610d84edfa8b4dcb52475e7fb6ae8321a5356e77b322666968f1705d8490c7df"} Nov 02 19:52:59.887205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:59.887171 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" event={"ID":"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3","Type":"ContainerStarted","Data":"c940ac51587461a5cfbf80782837f904649fb7e86f78378bd7629f34988a1ed6"} Nov 02 19:52:59.890132 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:59.890073 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" event={"ID":"49613984-cc5e-4db0-ba7d-2a6d321f7033","Type":"ContainerStarted","Data":"3df35cd5a7080000a768fb62b86defa64adff22b6fec418fa4024753092332a1"} Nov 02 19:52:59.893085 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:52:59.893057 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" event={"ID":"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7","Type":"ContainerStarted","Data":"5d31156a3eb76d16e4f4c103cd7225295359e93641615162f23f686ed2383308"} Nov 02 19:53:00.025356 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.025297 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" podStartSLOduration=2.720522734 podStartE2EDuration="26.025263625s" totalImagesPullingTime="23.304740891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:00.024939825 +0000 UTC m=+278.541712413" watchObservedRunningTime="2026-11-02 19:53:00.025263625 +0000 UTC m=+278.542036211" Nov 02 19:53:00.241185 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.241067 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" podStartSLOduration=2.635226881 podStartE2EDuration="31.241050906s" totalImagesPullingTime="28.605824025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:00.133427425 +0000 UTC m=+278.650200011" watchObservedRunningTime="2026-11-02 19:53:00.241050906 +0000 UTC m=+278.757823490" Nov 02 19:53:00.241559 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.241503 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" podStartSLOduration=3.364118049 podStartE2EDuration="26.241490916s" totalImagesPullingTime="22.877372867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:00.240513888 +0000 UTC m=+278.757286468" watchObservedRunningTime="2026-11-02 19:53:00.241490916 +0000 UTC m=+278.758263502" Nov 02 19:53:00.349182 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.349121 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" podStartSLOduration=3.5719584060000003 podStartE2EDuration="26.349100365s" totalImagesPullingTime="22.777141959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:00.34748741 +0000 UTC m=+278.864259997" watchObservedRunningTime="2026-11-02 19:53:00.349100365 +0000 UTC m=+278.865872950" Nov 02 19:53:00.899902 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.899855 5800 generic.go:356] "Generic (PLEG): container finished" podID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerID="3df35cd5a7080000a768fb62b86defa64adff22b6fec418fa4024753092332a1" exitCode=0 Nov 02 19:53:00.900360 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.899912 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" event={"ID":"49613984-cc5e-4db0-ba7d-2a6d321f7033","Type":"ContainerDied","Data":"3df35cd5a7080000a768fb62b86defa64adff22b6fec418fa4024753092332a1"} Nov 02 19:53:00.902347 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.902319 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" event={"ID":"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1","Type":"ContainerStarted","Data":"b9e9157101ffc50029063ae91e3de2b52eeb38bf91873daac06b06d529199973"} Nov 02 19:53:00.904893 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.904869 5800 generic.go:356] "Generic (PLEG): container finished" podID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerID="5d31156a3eb76d16e4f4c103cd7225295359e93641615162f23f686ed2383308" exitCode=0 Nov 02 19:53:00.905009 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.904946 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" event={"ID":"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7","Type":"ContainerDied","Data":"5d31156a3eb76d16e4f4c103cd7225295359e93641615162f23f686ed2383308"} Nov 02 19:53:00.907273 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.907254 5800 generic.go:356] "Generic (PLEG): container finished" podID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerID="610d84edfa8b4dcb52475e7fb6ae8321a5356e77b322666968f1705d8490c7df" exitCode=0 Nov 02 19:53:00.907386 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.907337 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" event={"ID":"1faa1e1a-fcb8-421f-8cc6-744f6691c98b","Type":"ContainerDied","Data":"610d84edfa8b4dcb52475e7fb6ae8321a5356e77b322666968f1705d8490c7df"} Nov 02 19:53:00.909645 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.909623 5800 generic.go:356] "Generic (PLEG): container finished" podID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerID="c940ac51587461a5cfbf80782837f904649fb7e86f78378bd7629f34988a1ed6" exitCode=0 Nov 02 19:53:00.909746 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:00.909654 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" event={"ID":"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3","Type":"ContainerDied","Data":"c940ac51587461a5cfbf80782837f904649fb7e86f78378bd7629f34988a1ed6"} Nov 02 19:53:01.329059 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:01.329004 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" podStartSLOduration=3.629267817 podStartE2EDuration="27.328989008s" totalImagesPullingTime="23.699721191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:01.328193888 +0000 UTC m=+279.844966472" watchObservedRunningTime="2026-11-02 19:53:01.328989008 +0000 UTC m=+279.845761605" Nov 02 19:53:01.916019 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:01.915927 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" event={"ID":"60bdb00e-3492-46b5-83bd-4189210aa89d","Type":"ContainerStarted","Data":"6dd6f2bf51b8fcdaad5b70991f96d186f21ffeed1d5305f4ef4e0d08e6f5ccb1"} Nov 02 19:53:01.918184 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:01.918159 5800 generic.go:356] "Generic (PLEG): container finished" podID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerID="b9e9157101ffc50029063ae91e3de2b52eeb38bf91873daac06b06d529199973" exitCode=0 Nov 02 19:53:01.918326 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:01.918253 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" event={"ID":"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1","Type":"ContainerDied","Data":"b9e9157101ffc50029063ae91e3de2b52eeb38bf91873daac06b06d529199973"} Nov 02 19:53:02.025123 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.025076 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" podStartSLOduration=3.245418393 podStartE2EDuration="31.025062569s" totalImagesPullingTime="27.779644176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:52:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:02.024207095 +0000 UTC m=+280.540979680" watchObservedRunningTime="2026-11-02 19:53:02.025062569 +0000 UTC m=+280.541835153" Nov 02 19:53:02.076154 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.076121 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:53:02.136192 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.136166 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:53:02.156305 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.156270 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:53:02.174276 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.174257 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:53:02.190963 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.190935 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-util\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-util\") pod \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " Nov 02 19:53:02.191070 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.190991 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-bundle\") pod \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " Nov 02 19:53:02.191070 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.191051 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-kube-api-access-kp7v6\" (UniqueName: \"kubernetes.io/projected/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-kube-api-access-kp7v6\") pod \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\" (UID: \"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3\") " Nov 02 19:53:02.191544 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.191513 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-bundle" pod "9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" (UID: "9a7c6a17-2912-4abd-90a7-7ef1ff4901b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.193335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.193262 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-kube-api-access-kp7v6" pod "9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" (UID: "9a7c6a17-2912-4abd-90a7-7ef1ff4901b3"). InnerVolumeSpecName "kube-api-access-kp7v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:02.196729 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.196705 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-util" pod "9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" (UID: "9a7c6a17-2912-4abd-90a7-7ef1ff4901b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.291938 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.291899 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-bundle\") pod \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " Nov 02 19:53:02.291938 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.291941 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-bundle\") pod \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " Nov 02 19:53:02.292182 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.291968 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-bundle\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-bundle\") pod \"49613984-cc5e-4db0-ba7d-2a6d321f7033\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " Nov 02 19:53:02.292182 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292018 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-kube-api-access-6tkxt\" (UniqueName: \"kubernetes.io/projected/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-kube-api-access-6tkxt\") pod \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " Nov 02 19:53:02.292182 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292051 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-util\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-util\") pod \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " Nov 02 19:53:02.292182 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292112 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49613984-cc5e-4db0-ba7d-2a6d321f7033-kube-api-access-c9dvt\" (UniqueName: \"kubernetes.io/projected/49613984-cc5e-4db0-ba7d-2a6d321f7033-kube-api-access-c9dvt\") pod \"49613984-cc5e-4db0-ba7d-2a6d321f7033\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " Nov 02 19:53:02.292182 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292148 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-kube-api-access-874h6\" (UniqueName: \"kubernetes.io/projected/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-kube-api-access-874h6\") pod \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\" (UID: \"1faa1e1a-fcb8-421f-8cc6-744f6691c98b\") " Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292190 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-util\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-util\") pod \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\" (UID: \"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7\") " Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292246 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-util\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-util\") pod \"49613984-cc5e-4db0-ba7d-2a6d321f7033\" (UID: \"49613984-cc5e-4db0-ba7d-2a6d321f7033\") " Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292544 5800 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-util\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292568 5800 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292571 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-bundle" pod "49613984-cc5e-4db0-ba7d-2a6d321f7033" (UID: "49613984-cc5e-4db0-ba7d-2a6d321f7033"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292583 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp7v6\" (UniqueName: \"kubernetes.io/projected/9a7c6a17-2912-4abd-90a7-7ef1ff4901b3-kube-api-access-kp7v6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292647 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-bundle" pod "1faa1e1a-fcb8-421f-8cc6-744f6691c98b" (UID: "1faa1e1a-fcb8-421f-8cc6-744f6691c98b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.292910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.292678 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-bundle" pod "d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" (UID: "d6d74a24-b691-4d94-bf5f-7ad1f0c662f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.294715 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.294684 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-kube-api-access-6tkxt" pod "d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" (UID: "d6d74a24-b691-4d94-bf5f-7ad1f0c662f7"). InnerVolumeSpecName "kube-api-access-6tkxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:02.294839 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.294767 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49613984-cc5e-4db0-ba7d-2a6d321f7033-kube-api-access-c9dvt" pod "49613984-cc5e-4db0-ba7d-2a6d321f7033" (UID: "49613984-cc5e-4db0-ba7d-2a6d321f7033"). InnerVolumeSpecName "kube-api-access-c9dvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:02.294965 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.294942 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-kube-api-access-874h6" pod "1faa1e1a-fcb8-421f-8cc6-744f6691c98b" (UID: "1faa1e1a-fcb8-421f-8cc6-744f6691c98b"). InnerVolumeSpecName "kube-api-access-874h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:02.296859 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.296841 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-util" pod "49613984-cc5e-4db0-ba7d-2a6d321f7033" (UID: "49613984-cc5e-4db0-ba7d-2a6d321f7033"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.298799 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.298775 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-util" pod "1faa1e1a-fcb8-421f-8cc6-744f6691c98b" (UID: "1faa1e1a-fcb8-421f-8cc6-744f6691c98b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.298860 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.298803 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-util" pod "d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" (UID: "d6d74a24-b691-4d94-bf5f-7ad1f0c662f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:02.393522 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393492 5800 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393522 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393521 5800 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393530 5800 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393539 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tkxt\" (UniqueName: \"kubernetes.io/projected/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-kube-api-access-6tkxt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393549 5800 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-util\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393563 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9dvt\" (UniqueName: \"kubernetes.io/projected/49613984-cc5e-4db0-ba7d-2a6d321f7033-kube-api-access-c9dvt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393573 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-874h6\" (UniqueName: \"kubernetes.io/projected/1faa1e1a-fcb8-421f-8cc6-744f6691c98b-kube-api-access-874h6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393582 5800 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6d74a24-b691-4d94-bf5f-7ad1f0c662f7-util\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.393710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.393590 5800 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49613984-cc5e-4db0-ba7d-2a6d321f7033-util\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:02.923880 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.923856 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" Nov 02 19:53:02.924386 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.923846 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fv5xk" event={"ID":"49613984-cc5e-4db0-ba7d-2a6d321f7033","Type":"ContainerDied","Data":"bc8ac98b1c484bf4a50d9cffbb509166396d8670f39d1edf63d351862764b34b"} Nov 02 19:53:02.924386 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.924003 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc8ac98b1c484bf4a50d9cffbb509166396d8670f39d1edf63d351862764b34b" Nov 02 19:53:02.925909 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.925884 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" Nov 02 19:53:02.925909 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.925892 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c926rx" event={"ID":"d6d74a24-b691-4d94-bf5f-7ad1f0c662f7","Type":"ContainerDied","Data":"02d572cbbfd3a7502cc3d84c369204bc7e5c86022b489c02e68d4186a272bc7e"} Nov 02 19:53:02.926107 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.925933 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d572cbbfd3a7502cc3d84c369204bc7e5c86022b489c02e68d4186a272bc7e" Nov 02 19:53:02.934866 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.934846 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" Nov 02 19:53:02.935116 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.934841 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j45qz" event={"ID":"1faa1e1a-fcb8-421f-8cc6-744f6691c98b","Type":"ContainerDied","Data":"c2c0b67c6aaf124859678fc2d5c200d35141e517e287d3f70e78b22bcc9a2ea9"} Nov 02 19:53:02.935191 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.935174 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c0b67c6aaf124859678fc2d5c200d35141e517e287d3f70e78b22bcc9a2ea9" Nov 02 19:53:02.937335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.937309 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" event={"ID":"9a7c6a17-2912-4abd-90a7-7ef1ff4901b3","Type":"ContainerDied","Data":"5d81db5d37e45779c8b882d7d441054383768f060d37d13f1a6851fa61e9ea69"} Nov 02 19:53:02.937456 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.937350 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d81db5d37e45779c8b882d7d441054383768f060d37d13f1a6851fa61e9ea69" Nov 02 19:53:02.937456 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.937354 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bllvh" Nov 02 19:53:02.939815 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.939794 5800 generic.go:356] "Generic (PLEG): container finished" podID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerID="6dd6f2bf51b8fcdaad5b70991f96d186f21ffeed1d5305f4ef4e0d08e6f5ccb1" exitCode=0 Nov 02 19:53:02.939944 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:02.939926 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" event={"ID":"60bdb00e-3492-46b5-83bd-4189210aa89d","Type":"ContainerDied","Data":"6dd6f2bf51b8fcdaad5b70991f96d186f21ffeed1d5305f4ef4e0d08e6f5ccb1"} Nov 02 19:53:03.086426 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.086397 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:53:03.199088 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.199006 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-kube-api-access-l4z54\" (UniqueName: \"kubernetes.io/projected/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-kube-api-access-l4z54\") pod \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " Nov 02 19:53:03.199088 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.199075 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-util\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-util\") pod \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " Nov 02 19:53:03.199345 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.199117 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-bundle\") pod \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\" (UID: \"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1\") " Nov 02 19:53:03.199814 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.199784 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-bundle" pod "cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" (UID: "cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:03.201246 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.201220 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-kube-api-access-l4z54" pod "cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" (UID: "cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1"). InnerVolumeSpecName "kube-api-access-l4z54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:03.204689 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.204670 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-util" pod "cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" (UID: "cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:03.300733 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.300696 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4z54\" (UniqueName: \"kubernetes.io/projected/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-kube-api-access-l4z54\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:03.300733 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.300726 5800 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-util\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:03.300733 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.300735 5800 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:03.946201 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.946119 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" Nov 02 19:53:03.946201 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.946150 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4tj4d" event={"ID":"cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1","Type":"ContainerDied","Data":"dd8e5888305f6c11697d26d6194abb9a51a52517d4b023abedf39516ad5f1354"} Nov 02 19:53:03.946201 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:03.946202 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd8e5888305f6c11697d26d6194abb9a51a52517d4b023abedf39516ad5f1354" Nov 02 19:53:04.074083 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.074059 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:53:04.208926 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.208841 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-util\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-util\") pod \"60bdb00e-3492-46b5-83bd-4189210aa89d\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " Nov 02 19:53:04.208926 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.208917 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-bundle\") pod \"60bdb00e-3492-46b5-83bd-4189210aa89d\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " Nov 02 19:53:04.209139 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.208960 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bdb00e-3492-46b5-83bd-4189210aa89d-kube-api-access-lcthb\" (UniqueName: \"kubernetes.io/projected/60bdb00e-3492-46b5-83bd-4189210aa89d-kube-api-access-lcthb\") pod \"60bdb00e-3492-46b5-83bd-4189210aa89d\" (UID: \"60bdb00e-3492-46b5-83bd-4189210aa89d\") " Nov 02 19:53:04.209562 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.209529 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-bundle" pod "60bdb00e-3492-46b5-83bd-4189210aa89d" (UID: "60bdb00e-3492-46b5-83bd-4189210aa89d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:04.210976 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.210959 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bdb00e-3492-46b5-83bd-4189210aa89d-kube-api-access-lcthb" pod "60bdb00e-3492-46b5-83bd-4189210aa89d" (UID: "60bdb00e-3492-46b5-83bd-4189210aa89d"). InnerVolumeSpecName "kube-api-access-lcthb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:53:04.212636 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.212610 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-util" pod "60bdb00e-3492-46b5-83bd-4189210aa89d" (UID: "60bdb00e-3492-46b5-83bd-4189210aa89d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:53:04.309578 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.309536 5800 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-util\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:04.309578 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.309572 5800 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60bdb00e-3492-46b5-83bd-4189210aa89d-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:04.309578 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.309585 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcthb\" (UniqueName: \"kubernetes.io/projected/60bdb00e-3492-46b5-83bd-4189210aa89d-kube-api-access-lcthb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:53:04.951591 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.951558 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" Nov 02 19:53:04.952080 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.951593 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbh279" event={"ID":"60bdb00e-3492-46b5-83bd-4189210aa89d","Type":"ContainerDied","Data":"615c202fdfa8e28ce24e3801878a97e24564f336cca0087fd498bfd10135fed2"} Nov 02 19:53:04.952080 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:04.951637 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="615c202fdfa8e28ce24e3801878a97e24564f336cca0087fd498bfd10135fed2" Nov 02 19:53:13.887836 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.887798 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl"] Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888099 5800 cpu_manager.go:414] "containerMap: removing container" podUID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerName="pull" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888110 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerName="pull" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888126 5800 cpu_manager.go:414] "containerMap: removing container" podUID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888131 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888141 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerName="pull" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888145 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerName="pull" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888155 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888160 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888166 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerName="extract" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888170 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerName="extract" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888176 5800 cpu_manager.go:414] "containerMap: removing container" podUID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerName="extract" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888181 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerName="extract" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888188 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888192 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888199 5800 cpu_manager.go:414] "containerMap: removing container" podUID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888204 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerName="util" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888210 5800 cpu_manager.go:414] "containerMap: removing container" podUID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerName="pull" Nov 02 19:53:13.888205 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888215 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerName="pull" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888223 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerName="pull" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888227 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerName="pull" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888234 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerName="util" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888238 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerName="util" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888244 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerName="pull" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888248 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerName="pull" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888254 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888258 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888263 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888267 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888276 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerName="util" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888294 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerName="util" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888300 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888305 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888315 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerName="pull" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888319 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerName="pull" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888324 5800 cpu_manager.go:414] "containerMap: removing container" podUID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888328 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888381 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="49613984-cc5e-4db0-ba7d-2a6d321f7033" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888393 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb146c37-4fbd-46f1-8bfb-b9eb3c88bba1" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888398 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bdb00e-3492-46b5-83bd-4189210aa89d" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888405 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7c6a17-2912-4abd-90a7-7ef1ff4901b3" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888411 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faa1e1a-fcb8-421f-8cc6-744f6691c98b" containerName="extract" Nov 02 19:53:13.888919 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.888417 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d74a24-b691-4d94-bf5f-7ad1f0c662f7" containerName="extract" Nov 02 19:53:13.891714 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.891694 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:13.945589 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.945558 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 19:53:13.945742 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.945594 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 19:53:13.945819 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.945806 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 19:53:13.945860 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.945814 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4tm9h\"" Nov 02 19:53:13.945860 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.945806 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 19:53:13.946040 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.946027 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 19:53:13.991473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.991440 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:13.991640 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.991506 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:13.991640 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.991612 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:13.991750 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.991655 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:13.991750 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.991683 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnv84\" (UniqueName: \"kubernetes.io/projected/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-kube-api-access-lnv84\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:13.991876 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.991828 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:13.994575 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:13.994552 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl"] Nov 02 19:53:14.092446 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.092413 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.092633 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.092463 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.092633 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.092491 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.092633 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.092511 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnv84\" (UniqueName: \"kubernetes.io/projected/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-kube-api-access-lnv84\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.092633 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.092537 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.092633 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.092559 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.092967 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.092941 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-manifests\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.094838 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.094813 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-tmp\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.094987 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.094970 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.095179 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.095156 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-metrics-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.095220 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.095161 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.149312 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.149220 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnv84\" (UniqueName: \"kubernetes.io/projected/f46c9a74-3aeb-4bb9-9df3-6a73955fc68c-kube-api-access-lnv84\") pod \"opendatahub-operator-controller-manager-55d64b46df-674hl\" (UID: \"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.206375 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.206342 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:14.673905 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.673878 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl"] Nov 02 19:53:14.676082 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:14.676057 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46c9a74_3aeb_4bb9_9df3_6a73955fc68c.slice/crio-a331c420acdfca22e02c15909955cb756dd51f1ad96d3e46f611a0420fb2b9df WatchSource:0}: Error finding container a331c420acdfca22e02c15909955cb756dd51f1ad96d3e46f611a0420fb2b9df: Status 404 returned error can't find the container with id a331c420acdfca22e02c15909955cb756dd51f1ad96d3e46f611a0420fb2b9df Nov 02 19:53:14.996812 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:14.996724 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" event={"ID":"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c","Type":"ContainerStarted","Data":"a331c420acdfca22e02c15909955cb756dd51f1ad96d3e46f611a0420fb2b9df"} Nov 02 19:53:16.865468 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.865430 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4"] Nov 02 19:53:16.871668 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.871647 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:16.914629 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.914597 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4"] Nov 02 19:53:16.914909 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.914883 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a3df6f-811e-4fc4-bfef-6753da0a89c7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4\" (UID: \"67a3df6f-811e-4fc4-bfef-6753da0a89c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:16.915006 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.914990 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpmx\" (UniqueName: \"kubernetes.io/projected/67a3df6f-811e-4fc4-bfef-6753da0a89c7-kube-api-access-nqpmx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4\" (UID: \"67a3df6f-811e-4fc4-bfef-6753da0a89c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:16.959392 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.959357 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:53:16.959568 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.959402 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 19:53:16.959618 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:16.959583 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qpglw\"" Nov 02 19:53:17.015431 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:17.015390 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a3df6f-811e-4fc4-bfef-6753da0a89c7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4\" (UID: \"67a3df6f-811e-4fc4-bfef-6753da0a89c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:17.015595 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:17.015468 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpmx\" (UniqueName: \"kubernetes.io/projected/67a3df6f-811e-4fc4-bfef-6753da0a89c7-kube-api-access-nqpmx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4\" (UID: \"67a3df6f-811e-4fc4-bfef-6753da0a89c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:17.015745 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:17.015718 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a3df6f-811e-4fc4-bfef-6753da0a89c7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4\" (UID: \"67a3df6f-811e-4fc4-bfef-6753da0a89c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:17.078057 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:17.078021 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpmx\" (UniqueName: \"kubernetes.io/projected/67a3df6f-811e-4fc4-bfef-6753da0a89c7-kube-api-access-nqpmx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4\" (UID: \"67a3df6f-811e-4fc4-bfef-6753da0a89c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:17.186646 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:17.186566 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" Nov 02 19:53:17.712843 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:17.712794 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a3df6f_811e_4fc4_bfef_6753da0a89c7.slice/crio-bd93255144e0db82a089ee89e77e9b6193affabaff34e53e3c9ba274b5228825 WatchSource:0}: Error finding container bd93255144e0db82a089ee89e77e9b6193affabaff34e53e3c9ba274b5228825: Status 404 returned error can't find the container with id bd93255144e0db82a089ee89e77e9b6193affabaff34e53e3c9ba274b5228825 Nov 02 19:53:17.714440 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:17.714412 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4"] Nov 02 19:53:18.015356 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:18.015314 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" event={"ID":"67a3df6f-811e-4fc4-bfef-6753da0a89c7","Type":"ContainerStarted","Data":"bd93255144e0db82a089ee89e77e9b6193affabaff34e53e3c9ba274b5228825"} Nov 02 19:53:19.021932 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:19.021899 5800 generic.go:356] "Generic (PLEG): container finished" podID="f46c9a74-3aeb-4bb9-9df3-6a73955fc68c" containerID="ed30e00ded50f0fe30b0146c5172c2b47f3cf98e8235e431e6d66ac963908ed5" exitCode=0 Nov 02 19:53:19.022429 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:19.021984 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" event={"ID":"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c","Type":"ContainerDied","Data":"ed30e00ded50f0fe30b0146c5172c2b47f3cf98e8235e431e6d66ac963908ed5"} Nov 02 19:53:20.029679 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:20.029649 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" event={"ID":"f46c9a74-3aeb-4bb9-9df3-6a73955fc68c","Type":"ContainerStarted","Data":"48808c3ce6fc4c06da63b54356a5491a87fead875c0a92f2a83483d96468358d"} Nov 02 19:53:20.030412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:20.030380 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:20.249478 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:20.249368 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" podStartSLOduration=2.244355315 podStartE2EDuration="7.249348666s" totalImagesPullingTime="4.541867488s" totalInitContainerRuntime="463.125863ms" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:20.202517534 +0000 UTC m=+298.719290120" watchObservedRunningTime="2026-11-02 19:53:20.249348666 +0000 UTC m=+298.766121248" Nov 02 19:53:21.036350 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:21.036311 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" event={"ID":"67a3df6f-811e-4fc4-bfef-6753da0a89c7","Type":"ContainerStarted","Data":"4dd11db5ea52783b15750d951eec8c356e9277daff2c144195ed7a460f14ff76"} Nov 02 19:53:21.158966 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:21.158867 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vl2s4" podStartSLOduration=2.022455178 podStartE2EDuration="5.158851594s" totalImagesPullingTime="3.136396416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:21.157103052 +0000 UTC m=+299.673875639" watchObservedRunningTime="2026-11-02 19:53:21.158851594 +0000 UTC m=+299.675624179" Nov 02 19:53:21.650229 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:21.650193 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:53:21.650712 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:21.650690 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:53:22.699871 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:53:22.699831 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:53:24.317525 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.317493 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf"] Nov 02 19:53:24.324032 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.324006 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:24.374126 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.374093 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57889d02-34f4-44fe-b4a7-6c50b09422c6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w2tkf\" (UID: \"57889d02-34f4-44fe-b4a7-6c50b09422c6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:24.374315 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.374243 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctq2\" (UniqueName: \"kubernetes.io/projected/57889d02-34f4-44fe-b4a7-6c50b09422c6-kube-api-access-6ctq2\") pod \"cert-manager-cainjector-5c57bbd4fc-w2tkf\" (UID: \"57889d02-34f4-44fe-b4a7-6c50b09422c6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:24.374583 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.374563 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf"] Nov 02 19:53:24.419386 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.419349 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 19:53:24.419386 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.419356 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 19:53:24.419589 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.419381 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4sqfw\"" Nov 02 19:53:24.475097 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.475065 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57889d02-34f4-44fe-b4a7-6c50b09422c6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w2tkf\" (UID: \"57889d02-34f4-44fe-b4a7-6c50b09422c6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:24.475291 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.475127 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctq2\" (UniqueName: \"kubernetes.io/projected/57889d02-34f4-44fe-b4a7-6c50b09422c6-kube-api-access-6ctq2\") pod \"cert-manager-cainjector-5c57bbd4fc-w2tkf\" (UID: \"57889d02-34f4-44fe-b4a7-6c50b09422c6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:24.527847 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.527815 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57889d02-34f4-44fe-b4a7-6c50b09422c6-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w2tkf\" (UID: \"57889d02-34f4-44fe-b4a7-6c50b09422c6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:24.528016 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.527997 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctq2\" (UniqueName: \"kubernetes.io/projected/57889d02-34f4-44fe-b4a7-6c50b09422c6-kube-api-access-6ctq2\") pod \"cert-manager-cainjector-5c57bbd4fc-w2tkf\" (UID: \"57889d02-34f4-44fe-b4a7-6c50b09422c6\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:24.646987 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:24.646910 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" Nov 02 19:53:25.012041 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:25.012016 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf"] Nov 02 19:53:25.014549 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:25.014517 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57889d02_34f4_44fe_b4a7_6c50b09422c6.slice/crio-3b78956ebbeb996439a2190e31c2c55ba86366ca044ee427a6949112a4b00b61 WatchSource:0}: Error finding container 3b78956ebbeb996439a2190e31c2c55ba86366ca044ee427a6949112a4b00b61: Status 404 returned error can't find the container with id 3b78956ebbeb996439a2190e31c2c55ba86366ca044ee427a6949112a4b00b61 Nov 02 19:53:25.016597 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:25.016574 5800 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:53:25.056589 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:25.056558 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" event={"ID":"57889d02-34f4-44fe-b4a7-6c50b09422c6","Type":"ContainerStarted","Data":"3b78956ebbeb996439a2190e31c2c55ba86366ca044ee427a6949112a4b00b61"} Nov 02 19:53:28.967977 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:28.967941 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x"] Nov 02 19:53:28.971128 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:28.971107 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.011839 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.011810 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8vc\" (UniqueName: \"kubernetes.io/projected/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-kube-api-access-zk8vc\") pod \"kuadrant-operator-controller-manager-cf8469d7b-jcw9x\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.011943 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.011845 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-jcw9x\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.024254 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.024231 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 19:53:29.024254 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.024234 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 19:53:29.025687 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.025670 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qcrlr\"" Nov 02 19:53:29.069790 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.069762 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x"] Nov 02 19:53:29.077769 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.077737 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" event={"ID":"57889d02-34f4-44fe-b4a7-6c50b09422c6","Type":"ContainerStarted","Data":"7670de49a7f20e2d0328d1f6255ba8c243dbf522b9faa4b435c6aa5935ab4ca2"} Nov 02 19:53:29.113267 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.113230 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8vc\" (UniqueName: \"kubernetes.io/projected/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-kube-api-access-zk8vc\") pod \"kuadrant-operator-controller-manager-cf8469d7b-jcw9x\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.113454 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.113272 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-jcw9x\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.113636 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.113618 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-jcw9x\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.167485 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.167453 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8vc\" (UniqueName: \"kubernetes.io/projected/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-kube-api-access-zk8vc\") pod \"kuadrant-operator-controller-manager-cf8469d7b-jcw9x\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.202439 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.202386 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2tkf" podStartSLOduration=1.239015723 podStartE2EDuration="5.202367502s" totalImagesPullingTime="3.963351779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:29.201051448 +0000 UTC m=+307.717824030" watchObservedRunningTime="2026-11-02 19:53:29.202367502 +0000 UTC m=+307.719140088" Nov 02 19:53:29.285557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.285524 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:29.702181 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:29.702155 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x"] Nov 02 19:53:29.703937 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:29.703906 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a8c0d2_5416_4a54_bf69_13d5a32a1b44.slice/crio-55480390411adfc19f6156dcf3460bf88a4580d259d3910bdebfc9c08fc1c399 WatchSource:0}: Error finding container 55480390411adfc19f6156dcf3460bf88a4580d259d3910bdebfc9c08fc1c399: Status 404 returned error can't find the container with id 55480390411adfc19f6156dcf3460bf88a4580d259d3910bdebfc9c08fc1c399 Nov 02 19:53:30.083684 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:30.083644 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" event={"ID":"24a8c0d2-5416-4a54-bf69-13d5a32a1b44","Type":"ContainerStarted","Data":"55480390411adfc19f6156dcf3460bf88a4580d259d3910bdebfc9c08fc1c399"} Nov 02 19:53:32.044532 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:32.044496 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55d64b46df-674hl" Nov 02 19:53:36.118631 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:36.118589 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" event={"ID":"24a8c0d2-5416-4a54-bf69-13d5a32a1b44","Type":"ContainerStarted","Data":"3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321"} Nov 02 19:53:36.119034 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:36.118677 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:36.323592 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:36.323538 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" podStartSLOduration=2.838384289 podStartE2EDuration="8.323523966s" totalImagesPullingTime="5.485139677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:36.276573966 +0000 UTC m=+314.793346548" watchObservedRunningTime="2026-11-02 19:53:36.323523966 +0000 UTC m=+314.840296550" Nov 02 19:53:37.070711 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.070671 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2mk96"] Nov 02 19:53:37.074359 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.074329 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" Nov 02 19:53:37.172409 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.172377 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t97gv\"" Nov 02 19:53:37.173709 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.173688 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2mk96"] Nov 02 19:53:37.187764 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.187739 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7jm\" (UniqueName: \"kubernetes.io/projected/cbda65e8-653a-4cf7-8e99-fecf2db77fd9-kube-api-access-rc7jm\") pod \"authorino-operator-7b7bb4bc77-2mk96\" (UID: \"cbda65e8-653a-4cf7-8e99-fecf2db77fd9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" Nov 02 19:53:37.288171 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.288135 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7jm\" (UniqueName: \"kubernetes.io/projected/cbda65e8-653a-4cf7-8e99-fecf2db77fd9-kube-api-access-rc7jm\") pod \"authorino-operator-7b7bb4bc77-2mk96\" (UID: \"cbda65e8-653a-4cf7-8e99-fecf2db77fd9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" Nov 02 19:53:37.341386 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.341317 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7jm\" (UniqueName: \"kubernetes.io/projected/cbda65e8-653a-4cf7-8e99-fecf2db77fd9-kube-api-access-rc7jm\") pod \"authorino-operator-7b7bb4bc77-2mk96\" (UID: \"cbda65e8-653a-4cf7-8e99-fecf2db77fd9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" Nov 02 19:53:37.387191 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.387159 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" Nov 02 19:53:37.759043 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:37.759014 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-2mk96"] Nov 02 19:53:37.761027 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:37.761000 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbda65e8_653a_4cf7_8e99_fecf2db77fd9.slice/crio-c9de0a14557d009e22503afc586022d7febcbebfee5201b3486a0a74ed1464e0 WatchSource:0}: Error finding container c9de0a14557d009e22503afc586022d7febcbebfee5201b3486a0a74ed1464e0: Status 404 returned error can't find the container with id c9de0a14557d009e22503afc586022d7febcbebfee5201b3486a0a74ed1464e0 Nov 02 19:53:38.129544 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:38.129452 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" event={"ID":"cbda65e8-653a-4cf7-8e99-fecf2db77fd9","Type":"ContainerStarted","Data":"c9de0a14557d009e22503afc586022d7febcbebfee5201b3486a0a74ed1464e0"} Nov 02 19:53:41.145119 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:41.145081 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" event={"ID":"cbda65e8-653a-4cf7-8e99-fecf2db77fd9","Type":"ContainerStarted","Data":"bce4bab33377576f95e2424faf5c391b2a41d55b91ddd22666b5b75bcd65196c"} Nov 02 19:53:41.145542 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:41.145203 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" Nov 02 19:53:41.257392 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:41.257334 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" podStartSLOduration=1.6719333870000002 podStartE2EDuration="4.257317254s" totalImagesPullingTime="2.585383867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:41.255944947 +0000 UTC m=+319.772717533" watchObservedRunningTime="2026-11-02 19:53:41.257317254 +0000 UTC m=+319.774089840" Nov 02 19:53:47.125720 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.125636 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:53:47.612976 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.612934 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k"] Nov 02 19:53:47.617655 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.617580 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:47.683006 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.682971 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxpn\" (UniqueName: \"kubernetes.io/projected/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-kube-api-access-sdxpn\") pod \"kuadrant-operator-controller-manager-678d87766f-vnq4k\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:47.683195 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.683083 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vnq4k\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:47.709152 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.709112 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k"] Nov 02 19:53:47.783864 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.783829 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vnq4k\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:47.784049 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.783940 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxpn\" (UniqueName: \"kubernetes.io/projected/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-kube-api-access-sdxpn\") pod \"kuadrant-operator-controller-manager-678d87766f-vnq4k\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:47.784230 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.784207 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-vnq4k\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:47.837479 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.837452 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxpn\" (UniqueName: \"kubernetes.io/projected/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-kube-api-access-sdxpn\") pod \"kuadrant-operator-controller-manager-678d87766f-vnq4k\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:47.934249 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:47.934170 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:48.347759 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:48.347721 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k"] Nov 02 19:53:48.351064 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:48.351037 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a2f253_46f8_4c5f_ad82_dcd0bac32f1b.slice/crio-363b86718a3d894ecea6d690a503f97ae3606047b412fc58bff7935103676cfd WatchSource:0}: Error finding container 363b86718a3d894ecea6d690a503f97ae3606047b412fc58bff7935103676cfd: Status 404 returned error can't find the container with id 363b86718a3d894ecea6d690a503f97ae3606047b412fc58bff7935103676cfd Nov 02 19:53:49.187057 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:49.187015 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" event={"ID":"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b","Type":"ContainerStarted","Data":"0e4656216550d4af0ca6513f65de98680dc0f0fe58434c33fe122ee6a691e5d4"} Nov 02 19:53:49.187253 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:49.187081 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:53:49.187253 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:49.187098 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" event={"ID":"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b","Type":"ContainerStarted","Data":"363b86718a3d894ecea6d690a503f97ae3606047b412fc58bff7935103676cfd"} Nov 02 19:53:49.406034 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:49.405950 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" podStartSLOduration=2.405935088 podStartE2EDuration="2.405935088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:49.35766541 +0000 UTC m=+327.874437995" watchObservedRunningTime="2026-11-02 19:53:49.405935088 +0000 UTC m=+327.922707673" Nov 02 19:53:50.512863 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.512828 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl"] Nov 02 19:53:50.519016 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.518996 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.568164 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.568136 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4n4mw\"" Nov 02 19:53:50.612079 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612035 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612243 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612091 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f35ad8-9615-4438-9f8d-5f2141131f13-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612243 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612132 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612243 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612219 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612403 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612246 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612403 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612272 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6r7\" (UniqueName: \"kubernetes.io/projected/14f35ad8-9615-4438-9f8d-5f2141131f13-kube-api-access-ht6r7\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612403 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612360 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612552 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612533 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.612636 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.612616 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.616771 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.616747 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl"] Nov 02 19:53:50.713203 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713162 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.713399 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713215 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.713399 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713241 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.713399 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713267 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6r7\" (UniqueName: \"kubernetes.io/projected/14f35ad8-9615-4438-9f8d-5f2141131f13-kube-api-access-ht6r7\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.713524 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713415 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.713569 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713520 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.713620 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713604 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.714086 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713628 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.714086 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713647 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.714086 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713680 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.714086 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713689 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f35ad8-9615-4438-9f8d-5f2141131f13-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.714086 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713716 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.714086 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.713986 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.714448 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.714333 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f35ad8-9615-4438-9f8d-5f2141131f13-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.715945 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.715918 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.716125 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.716104 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.772256 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.772182 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f35ad8-9615-4438-9f8d-5f2141131f13-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.772725 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.772706 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6r7\" (UniqueName: \"kubernetes.io/projected/14f35ad8-9615-4438-9f8d-5f2141131f13-kube-api-access-ht6r7\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b579bndl\" (UID: \"14f35ad8-9615-4438-9f8d-5f2141131f13\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:50.831872 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:50.831841 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:51.253995 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:51.253963 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl"] Nov 02 19:53:51.256333 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:51.256306 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f35ad8_9615_4438_9f8d_5f2141131f13.slice/crio-c5ea8325d56409e75ebc68958275b070ac0c943ef3dae799941e3090d35f1ec1 WatchSource:0}: Error finding container c5ea8325d56409e75ebc68958275b070ac0c943ef3dae799941e3090d35f1ec1: Status 404 returned error can't find the container with id c5ea8325d56409e75ebc68958275b070ac0c943ef3dae799941e3090d35f1ec1 Nov 02 19:53:51.258479 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:51.258451 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:53:51.258559 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:51.258515 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:53:51.258559 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:51.258545 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:53:52.152469 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:52.152438 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-2mk96" Nov 02 19:53:52.203988 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:52.203950 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" event={"ID":"14f35ad8-9615-4438-9f8d-5f2141131f13","Type":"ContainerStarted","Data":"61f3ac07c99d44fa6c467d0c5d1214ea5ecbdbbf9e2282e9e5d88a41dcb4d5c5"} Nov 02 19:53:52.204145 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:52.204014 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" event={"ID":"14f35ad8-9615-4438-9f8d-5f2141131f13","Type":"ContainerStarted","Data":"c5ea8325d56409e75ebc68958275b070ac0c943ef3dae799941e3090d35f1ec1"} Nov 02 19:53:52.376120 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:52.376060 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" podStartSLOduration=2.376042256 podStartE2EDuration="2.376042256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:53:52.373691145 +0000 UTC m=+330.890463731" watchObservedRunningTime="2026-11-02 19:53:52.376042256 +0000 UTC m=+330.892814841" Nov 02 19:53:52.696251 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:53:52.696214 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:53:52.833015 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:52.832978 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:52.837898 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:52.837873 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:53.209452 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:53.209418 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:53.210587 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:53.210570 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b579bndl" Nov 02 19:53:56.281881 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.281848 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mpscz"] Nov 02 19:53:56.288893 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.288867 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" Nov 02 19:53:56.336678 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.336640 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mpscz"] Nov 02 19:53:56.369923 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.369890 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c67k\" (UniqueName: \"kubernetes.io/projected/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04-kube-api-access-7c67k\") pod \"authorino-56c6c9ddd5-mpscz\" (UID: \"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04\") " pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" Nov 02 19:53:56.381100 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.381072 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vvsx7\"" Nov 02 19:53:56.470959 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.470923 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c67k\" (UniqueName: \"kubernetes.io/projected/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04-kube-api-access-7c67k\") pod \"authorino-56c6c9ddd5-mpscz\" (UID: \"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04\") " pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" Nov 02 19:53:56.524754 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.524719 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c67k\" (UniqueName: \"kubernetes.io/projected/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04-kube-api-access-7c67k\") pod \"authorino-56c6c9ddd5-mpscz\" (UID: \"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04\") " pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" Nov 02 19:53:56.602084 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.601994 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" Nov 02 19:53:56.967362 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:56.967325 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mpscz"] Nov 02 19:53:56.970980 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:53:56.970943 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda85e1ff1_e844_4b9a_a4cb_8505afbf1c04.slice/crio-a67246e561e0e3bf614a796eb541f59500016e0afe3f066ab3355afdefda9c82 WatchSource:0}: Error finding container a67246e561e0e3bf614a796eb541f59500016e0afe3f066ab3355afdefda9c82: Status 404 returned error can't find the container with id a67246e561e0e3bf614a796eb541f59500016e0afe3f066ab3355afdefda9c82 Nov 02 19:53:57.230518 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:53:57.230433 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" event={"ID":"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04","Type":"ContainerStarted","Data":"a67246e561e0e3bf614a796eb541f59500016e0afe3f066ab3355afdefda9c82"} Nov 02 19:54:00.195425 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.194763 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 19:54:00.401897 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.401849 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x"] Nov 02 19:54:00.402444 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.402406 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" podUID="24a8c0d2-5416-4a54-bf69-13d5a32a1b44" containerName="manager" containerID="cri-o://3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321" gracePeriod=10 Nov 02 19:54:00.698481 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.698387 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:54:00.809150 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.809069 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-kube-api-access-zk8vc\" (UniqueName: \"kubernetes.io/projected/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-kube-api-access-zk8vc\") pod \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " Nov 02 19:54:00.809150 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.809143 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-extensions-socket-volume\") pod \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\" (UID: \"24a8c0d2-5416-4a54-bf69-13d5a32a1b44\") " Nov 02 19:54:00.809769 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.809739 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-extensions-socket-volume" pod "24a8c0d2-5416-4a54-bf69-13d5a32a1b44" (UID: "24a8c0d2-5416-4a54-bf69-13d5a32a1b44"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 19:54:00.811547 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.811526 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-kube-api-access-zk8vc" pod "24a8c0d2-5416-4a54-bf69-13d5a32a1b44" (UID: "24a8c0d2-5416-4a54-bf69-13d5a32a1b44"). InnerVolumeSpecName "kube-api-access-zk8vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:54:00.909815 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.909782 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8vc\" (UniqueName: \"kubernetes.io/projected/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-kube-api-access-zk8vc\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:00.909815 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:00.909813 5800 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24a8c0d2-5416-4a54-bf69-13d5a32a1b44-extensions-socket-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:54:01.252352 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.252315 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" event={"ID":"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04","Type":"ContainerStarted","Data":"1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd"} Nov 02 19:54:01.253781 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.253757 5800 generic.go:356] "Generic (PLEG): container finished" podID="24a8c0d2-5416-4a54-bf69-13d5a32a1b44" containerID="3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321" exitCode=0 Nov 02 19:54:01.253883 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.253797 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" Nov 02 19:54:01.253883 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.253805 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" event={"ID":"24a8c0d2-5416-4a54-bf69-13d5a32a1b44","Type":"ContainerDied","Data":"3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321"} Nov 02 19:54:01.253883 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.253865 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x" event={"ID":"24a8c0d2-5416-4a54-bf69-13d5a32a1b44","Type":"ContainerDied","Data":"55480390411adfc19f6156dcf3460bf88a4580d259d3910bdebfc9c08fc1c399"} Nov 02 19:54:01.253999 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.253886 5800 scope.go:122] "RemoveContainer" containerID="3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321" Nov 02 19:54:01.263446 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.263429 5800 scope.go:122] "RemoveContainer" containerID="3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321" Nov 02 19:54:01.263698 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:54:01.263681 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321\": container with ID starting with 3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321 not found: ID does not exist" containerID="3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321" Nov 02 19:54:01.263759 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.263706 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321"} err="failed to get container status \"3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321\": rpc error: code = NotFound desc = could not find container \"3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321\": container with ID starting with 3dabb4a7ac3307ca055e44bf5badcee7d4941e3908db05d99a9d7aac2fd25321 not found: ID does not exist" Nov 02 19:54:01.358005 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.357933 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" podStartSLOduration=1.645856938 podStartE2EDuration="5.357917692s" totalImagesPullingTime="3.712060754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:01.357727052 +0000 UTC m=+339.874499639" watchObservedRunningTime="2026-11-02 19:54:01.357917692 +0000 UTC m=+339.874690278" Nov 02 19:54:01.461374 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.461342 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x"] Nov 02 19:54:01.507948 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.507855 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-jcw9x"] Nov 02 19:54:01.620105 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:01.620073 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a8c0d2-5416-4a54-bf69-13d5a32a1b44" path="/var/lib/kubelet/pods/24a8c0d2-5416-4a54-bf69-13d5a32a1b44/volumes" Nov 02 19:54:12.380724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.380684 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v"] Nov 02 19:54:12.381188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.381093 5800 cpu_manager.go:414] "containerMap: removing container" podUID="24a8c0d2-5416-4a54-bf69-13d5a32a1b44" containerName="manager" Nov 02 19:54:12.381188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.381106 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a8c0d2-5416-4a54-bf69-13d5a32a1b44" containerName="manager" Nov 02 19:54:12.381257 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.381191 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a8c0d2-5416-4a54-bf69-13d5a32a1b44" containerName="manager" Nov 02 19:54:12.387319 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.387273 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.408844 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.408818 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd928515-6b72-45ec-baea-ffd4ef599868-manager-config\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.408984 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.408856 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9d8p\" (UniqueName: \"kubernetes.io/projected/cd928515-6b72-45ec-baea-ffd4ef599868-kube-api-access-m9d8p\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.408984 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.408882 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd928515-6b72-45ec-baea-ffd4ef599868-cert\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.409060 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.409011 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd928515-6b72-45ec-baea-ffd4ef599868-metrics-certs\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.437341 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.437301 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 19:54:12.438059 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.438038 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v"] Nov 02 19:54:12.482202 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.482164 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 19:54:12.482202 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.482185 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 19:54:12.483693 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.483672 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-8jbxn\"" Nov 02 19:54:12.483754 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.483674 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 19:54:12.483754 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.483674 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 19:54:12.510223 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.510184 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd928515-6b72-45ec-baea-ffd4ef599868-manager-config\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.510435 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.510242 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9d8p\" (UniqueName: \"kubernetes.io/projected/cd928515-6b72-45ec-baea-ffd4ef599868-kube-api-access-m9d8p\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.510435 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.510269 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd928515-6b72-45ec-baea-ffd4ef599868-cert\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.510435 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.510310 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd928515-6b72-45ec-baea-ffd4ef599868-metrics-certs\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.511076 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.511046 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd928515-6b72-45ec-baea-ffd4ef599868-manager-config\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.513145 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.513122 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd928515-6b72-45ec-baea-ffd4ef599868-metrics-certs\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.513258 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.513167 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd928515-6b72-45ec-baea-ffd4ef599868-cert\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.564460 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.564429 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9d8p\" (UniqueName: \"kubernetes.io/projected/cd928515-6b72-45ec-baea-ffd4ef599868-kube-api-access-m9d8p\") pod \"jobset-controller-manager-558b5659f-7f55v\" (UID: \"cd928515-6b72-45ec-baea-ffd4ef599868\") " pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:12.699861 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:12.699774 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:13.064529 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:13.064499 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v"] Nov 02 19:54:13.067021 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:54:13.066991 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd928515_6b72_45ec_baea_ffd4ef599868.slice/crio-f40337b7edea29109b90f199ed4656b6e3da85034eceaf7484154d961801bb58 WatchSource:0}: Error finding container f40337b7edea29109b90f199ed4656b6e3da85034eceaf7484154d961801bb58: Status 404 returned error can't find the container with id f40337b7edea29109b90f199ed4656b6e3da85034eceaf7484154d961801bb58 Nov 02 19:54:13.318716 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:13.318625 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" event={"ID":"cd928515-6b72-45ec-baea-ffd4ef599868","Type":"ContainerStarted","Data":"f40337b7edea29109b90f199ed4656b6e3da85034eceaf7484154d961801bb58"} Nov 02 19:54:18.346523 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:18.346482 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" event={"ID":"cd928515-6b72-45ec-baea-ffd4ef599868","Type":"ContainerStarted","Data":"3295d5d41be35e891d33fa345b06795235fefe38a2349265c0dda272685ee2d2"} Nov 02 19:54:18.347010 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:18.346588 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:18.455774 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:18.455716 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" podStartSLOduration=2.039243225 podStartE2EDuration="6.45570042s" totalImagesPullingTime="4.416457195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:54:18.454218771 +0000 UTC m=+356.970991357" watchObservedRunningTime="2026-11-02 19:54:18.45570042 +0000 UTC m=+356.972473004" Nov 02 19:54:22.694959 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:54:22.694896 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:54:29.357131 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:54:29.357098 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-558b5659f-7f55v" Nov 02 19:54:52.694473 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:54:52.694429 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:55:22.695187 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:55:22.695144 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:55:36.739340 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:36.739218 5800 generic.go:356] "Generic (PLEG): container finished" podID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerID="934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004" exitCode=0 Nov 02 19:55:36.739340 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:36.739314 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" event={"ID":"5dfdfd09-0256-4ca4-a4d1-c99ec864c822","Type":"ContainerDied","Data":"934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004"} Nov 02 19:55:36.739781 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:36.739685 5800 scope.go:122] "RemoveContainer" containerID="934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004" Nov 02 19:55:36.740804 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:36.740777 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:55:36.740871 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:36.740849 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:55:36.740907 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:36.740885 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:55:37.757109 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:37.757076 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" event={"ID":"5dfdfd09-0256-4ca4-a4d1-c99ec864c822","Type":"ContainerStarted","Data":"143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949"} Nov 02 19:55:38.602986 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:38.602946 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:55:38.603178 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:38.603145 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-slhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 02 19:55:38.603223 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:38.603196 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 02 19:55:39.603237 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:39.603194 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-slhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 02 19:55:39.603770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:39.603263 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 02 19:55:39.983804 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:39.983726 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gzcpm"] Nov 02 19:55:39.987437 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:39.987416 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" Nov 02 19:55:40.042403 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.042364 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gzcpm"] Nov 02 19:55:40.085664 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.085631 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxf4\" (UniqueName: \"kubernetes.io/projected/4e4555df-4a18-4c08-93e2-2392d3e3416b-kube-api-access-cvxf4\") pod \"authorino-6f6944bcdd-gzcpm\" (UID: \"4e4555df-4a18-4c08-93e2-2392d3e3416b\") " pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" Nov 02 19:55:40.186646 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.186599 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxf4\" (UniqueName: \"kubernetes.io/projected/4e4555df-4a18-4c08-93e2-2392d3e3416b-kube-api-access-cvxf4\") pod \"authorino-6f6944bcdd-gzcpm\" (UID: \"4e4555df-4a18-4c08-93e2-2392d3e3416b\") " pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" Nov 02 19:55:40.253189 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.253116 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxf4\" (UniqueName: \"kubernetes.io/projected/4e4555df-4a18-4c08-93e2-2392d3e3416b-kube-api-access-cvxf4\") pod \"authorino-6f6944bcdd-gzcpm\" (UID: \"4e4555df-4a18-4c08-93e2-2392d3e3416b\") " pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" Nov 02 19:55:40.298913 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.298880 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" Nov 02 19:55:40.352336 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.337238 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gzcpm"] Nov 02 19:55:40.602696 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.602623 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-slhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 02 19:55:40.602696 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.602680 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 02 19:55:40.687742 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.687717 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gzcpm"] Nov 02 19:55:40.689734 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:55:40.689708 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e4555df_4a18_4c08_93e2_2392d3e3416b.slice/crio-9de4d920152c1c04e5aafeeb9a2cfa78bb3865a53487b10983b4d9a1248b0939 WatchSource:0}: Error finding container 9de4d920152c1c04e5aafeeb9a2cfa78bb3865a53487b10983b4d9a1248b0939: Status 404 returned error can't find the container with id 9de4d920152c1c04e5aafeeb9a2cfa78bb3865a53487b10983b4d9a1248b0939 Nov 02 19:55:40.773689 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.773648 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" event={"ID":"4e4555df-4a18-4c08-93e2-2392d3e3416b","Type":"ContainerStarted","Data":"9de4d920152c1c04e5aafeeb9a2cfa78bb3865a53487b10983b4d9a1248b0939"} Nov 02 19:55:40.817420 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.817393 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d9d749ccc-h4fh9"] Nov 02 19:55:40.821908 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.821887 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:40.871629 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.871563 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 19:55:40.872617 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.872595 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d9d749ccc-h4fh9"] Nov 02 19:55:40.892574 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.892549 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tvb4\" (UniqueName: \"kubernetes.io/projected/567e8076-1152-4132-8a42-3e3959dbdca2-kube-api-access-6tvb4\") pod \"authorino-d9d749ccc-h4fh9\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:40.892690 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.892600 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/567e8076-1152-4132-8a42-3e3959dbdca2-tls-cert\") pod \"authorino-d9d749ccc-h4fh9\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:40.993909 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.993873 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tvb4\" (UniqueName: \"kubernetes.io/projected/567e8076-1152-4132-8a42-3e3959dbdca2-kube-api-access-6tvb4\") pod \"authorino-d9d749ccc-h4fh9\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:40.994085 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.993937 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/567e8076-1152-4132-8a42-3e3959dbdca2-tls-cert\") pod \"authorino-d9d749ccc-h4fh9\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:40.996723 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:40.996699 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/567e8076-1152-4132-8a42-3e3959dbdca2-tls-cert\") pod \"authorino-d9d749ccc-h4fh9\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:41.054295 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:41.054246 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tvb4\" (UniqueName: \"kubernetes.io/projected/567e8076-1152-4132-8a42-3e3959dbdca2-kube-api-access-6tvb4\") pod \"authorino-d9d749ccc-h4fh9\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:41.156199 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:41.156112 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:55:41.521478 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:41.521443 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d9d749ccc-h4fh9"] Nov 02 19:55:41.524932 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:55:41.524902 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod567e8076_1152_4132_8a42_3e3959dbdca2.slice/crio-1c8c445fc7ea6d981398582c8a5c28d9cbcb7684d4fc2d5c9744436418a30b45 WatchSource:0}: Error finding container 1c8c445fc7ea6d981398582c8a5c28d9cbcb7684d4fc2d5c9744436418a30b45: Status 404 returned error can't find the container with id 1c8c445fc7ea6d981398582c8a5c28d9cbcb7684d4fc2d5c9744436418a30b45 Nov 02 19:55:41.602642 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:41.602599 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-slhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 02 19:55:41.602807 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:41.602656 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 02 19:55:41.780306 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:41.780200 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" event={"ID":"567e8076-1152-4132-8a42-3e3959dbdca2","Type":"ContainerStarted","Data":"1c8c445fc7ea6d981398582c8a5c28d9cbcb7684d4fc2d5c9744436418a30b45"} Nov 02 19:55:42.603507 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:42.603482 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-slhh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Nov 02 19:55:42.603677 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:42.603659 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Nov 02 19:55:42.789073 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:42.788976 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" event={"ID":"4e4555df-4a18-4c08-93e2-2392d3e3416b","Type":"ContainerStarted","Data":"066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2"} Nov 02 19:55:42.789073 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:42.789023 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" podUID="4e4555df-4a18-4c08-93e2-2392d3e3416b" containerName="authorino" containerID="cri-o://066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2" gracePeriod=30 Nov 02 19:55:42.894073 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:42.894022 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" podStartSLOduration=2.066907131 podStartE2EDuration="3.894006021s" totalImagesPullingTime="1.82709889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:42.892963318 +0000 UTC m=+441.409735900" watchObservedRunningTime="2026-11-02 19:55:42.894006021 +0000 UTC m=+441.410778621" Nov 02 19:55:43.026478 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.026452 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" Nov 02 19:55:43.113041 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.113007 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4555df-4a18-4c08-93e2-2392d3e3416b-kube-api-access-cvxf4\" (UniqueName: \"kubernetes.io/projected/4e4555df-4a18-4c08-93e2-2392d3e3416b-kube-api-access-cvxf4\") pod \"4e4555df-4a18-4c08-93e2-2392d3e3416b\" (UID: \"4e4555df-4a18-4c08-93e2-2392d3e3416b\") " Nov 02 19:55:43.115046 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.115019 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4555df-4a18-4c08-93e2-2392d3e3416b-kube-api-access-cvxf4" pod "4e4555df-4a18-4c08-93e2-2392d3e3416b" (UID: "4e4555df-4a18-4c08-93e2-2392d3e3416b"). InnerVolumeSpecName "kube-api-access-cvxf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:43.214344 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.214310 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxf4\" (UniqueName: \"kubernetes.io/projected/4e4555df-4a18-4c08-93e2-2392d3e3416b-kube-api-access-cvxf4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:43.607468 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.607425 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:55:43.607724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.607700 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:55:43.608511 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.608490 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 19:55:43.796829 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.796711 5800 generic.go:356] "Generic (PLEG): container finished" podID="4e4555df-4a18-4c08-93e2-2392d3e3416b" containerID="066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2" exitCode=0 Nov 02 19:55:43.796829 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.796757 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" event={"ID":"4e4555df-4a18-4c08-93e2-2392d3e3416b","Type":"ContainerDied","Data":"066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2"} Nov 02 19:55:43.796829 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.796812 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" Nov 02 19:55:43.796829 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.796821 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gzcpm" event={"ID":"4e4555df-4a18-4c08-93e2-2392d3e3416b","Type":"ContainerDied","Data":"9de4d920152c1c04e5aafeeb9a2cfa78bb3865a53487b10983b4d9a1248b0939"} Nov 02 19:55:43.797476 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.796849 5800 scope.go:122] "RemoveContainer" containerID="066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2" Nov 02 19:55:43.802120 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.802087 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" event={"ID":"567e8076-1152-4132-8a42-3e3959dbdca2","Type":"ContainerStarted","Data":"cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33"} Nov 02 19:55:43.814422 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.814358 5800 scope.go:122] "RemoveContainer" containerID="066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2" Nov 02 19:55:43.815438 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:55:43.815407 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2\": container with ID starting with 066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2 not found: ID does not exist" containerID="066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2" Nov 02 19:55:43.815574 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.815454 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2"} err="failed to get container status \"066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2\": rpc error: code = NotFound desc = could not find container \"066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2\": container with ID starting with 066f9be6503c2a44085576ff670da8763910e49eadfd2a2e5a0fbaa13b6e26c2 not found: ID does not exist" Nov 02 19:55:43.977794 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.977759 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gzcpm"] Nov 02 19:55:43.982104 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:43.982072 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gzcpm"] Nov 02 19:55:44.091163 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.091050 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" podStartSLOduration=2.679755676 podStartE2EDuration="4.091035241s" totalImagesPullingTime="1.411279565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:55:44.090044879 +0000 UTC m=+442.606817465" watchObservedRunningTime="2026-11-02 19:55:44.091035241 +0000 UTC m=+442.607807825" Nov 02 19:55:44.113461 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.113429 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mpscz"] Nov 02 19:55:44.113651 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.113630 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" podUID="a85e1ff1-e844-4b9a-a4cb-8505afbf1c04" containerName="authorino" containerID="cri-o://1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd" gracePeriod=30 Nov 02 19:55:44.368155 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.368128 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" Nov 02 19:55:44.430647 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.430617 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04-kube-api-access-7c67k\" (UniqueName: \"kubernetes.io/projected/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04-kube-api-access-7c67k\") pod \"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04\" (UID: \"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04\") " Nov 02 19:55:44.432792 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.432764 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04-kube-api-access-7c67k" pod "a85e1ff1-e844-4b9a-a4cb-8505afbf1c04" (UID: "a85e1ff1-e844-4b9a-a4cb-8505afbf1c04"). InnerVolumeSpecName "kube-api-access-7c67k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:55:44.532228 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.532194 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c67k\" (UniqueName: \"kubernetes.io/projected/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04-kube-api-access-7c67k\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:55:44.807588 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.807549 5800 generic.go:356] "Generic (PLEG): container finished" podID="a85e1ff1-e844-4b9a-a4cb-8505afbf1c04" containerID="1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd" exitCode=0 Nov 02 19:55:44.808029 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.807600 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" Nov 02 19:55:44.808029 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.807640 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" event={"ID":"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04","Type":"ContainerDied","Data":"1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd"} Nov 02 19:55:44.808029 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.807687 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mpscz" event={"ID":"a85e1ff1-e844-4b9a-a4cb-8505afbf1c04","Type":"ContainerDied","Data":"a67246e561e0e3bf614a796eb541f59500016e0afe3f066ab3355afdefda9c82"} Nov 02 19:55:44.808029 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.807710 5800 scope.go:122] "RemoveContainer" containerID="1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd" Nov 02 19:55:44.820452 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.820430 5800 scope.go:122] "RemoveContainer" containerID="1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd" Nov 02 19:55:44.820734 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:55:44.820714 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd\": container with ID starting with 1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd not found: ID does not exist" containerID="1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd" Nov 02 19:55:44.820780 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.820744 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd"} err="failed to get container status \"1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd\": rpc error: code = NotFound desc = could not find container \"1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd\": container with ID starting with 1feeb917de25aaef7dd9f1bf548cdbb392f37926aa733b9af9b015e5b8b21dbd not found: ID does not exist" Nov 02 19:55:44.959006 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.958971 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mpscz"] Nov 02 19:55:44.963039 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:44.963014 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mpscz"] Nov 02 19:55:45.620800 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:45.620765 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4555df-4a18-4c08-93e2-2392d3e3416b" path="/var/lib/kubelet/pods/4e4555df-4a18-4c08-93e2-2392d3e3416b/volumes" Nov 02 19:55:45.621125 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:55:45.621112 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85e1ff1-e844-4b9a-a4cb-8505afbf1c04" path="/var/lib/kubelet/pods/a85e1ff1-e844-4b9a-a4cb-8505afbf1c04/volumes" Nov 02 19:55:52.695055 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:55:52.695011 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:56:22.696826 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:56:22.696782 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:56:52.697540 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:56:52.697492 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:57:09.374095 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.374059 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42ksz"] Nov 02 19:57:09.374497 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.374457 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a85e1ff1-e844-4b9a-a4cb-8505afbf1c04" containerName="authorino" Nov 02 19:57:09.374497 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.374469 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85e1ff1-e844-4b9a-a4cb-8505afbf1c04" containerName="authorino" Nov 02 19:57:09.374497 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.374484 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4e4555df-4a18-4c08-93e2-2392d3e3416b" containerName="authorino" Nov 02 19:57:09.374497 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.374489 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4555df-4a18-4c08-93e2-2392d3e3416b" containerName="authorino" Nov 02 19:57:09.374629 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.374542 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85e1ff1-e844-4b9a-a4cb-8505afbf1c04" containerName="authorino" Nov 02 19:57:09.374629 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.374552 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4555df-4a18-4c08-93e2-2392d3e3416b" containerName="authorino" Nov 02 19:57:09.377535 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.377517 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:09.395485 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.395462 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn979\" (UniqueName: \"kubernetes.io/projected/0cd9fdd9-fd6a-415a-9d89-655b5637a192-kube-api-access-gn979\") pod \"payload-pre-processing-7b98bb678c-42ksz\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:09.395594 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.395499 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd9fdd9-fd6a-415a-9d89-655b5637a192-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-42ksz\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:09.474432 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.474382 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 19:57:09.474432 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.474408 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-msppk\"" Nov 02 19:57:09.476227 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.476197 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42ksz"] Nov 02 19:57:09.496990 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.496949 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd9fdd9-fd6a-415a-9d89-655b5637a192-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-42ksz\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:09.497160 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.497142 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn979\" (UniqueName: \"kubernetes.io/projected/0cd9fdd9-fd6a-415a-9d89-655b5637a192-kube-api-access-gn979\") pod \"payload-pre-processing-7b98bb678c-42ksz\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:09.497687 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.497668 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd9fdd9-fd6a-415a-9d89-655b5637a192-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-42ksz\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:09.550793 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.550757 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn979\" (UniqueName: \"kubernetes.io/projected/0cd9fdd9-fd6a-415a-9d89-655b5637a192-kube-api-access-gn979\") pod \"payload-pre-processing-7b98bb678c-42ksz\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:09.691472 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:09.691386 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:10.052820 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:10.052794 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42ksz"] Nov 02 19:57:10.054805 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:57:10.054776 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd9fdd9_fd6a_415a_9d89_655b5637a192.slice/crio-6a65478cafc38553b344e21dd70113ce277574e11bc0c0008569736640733c62 WatchSource:0}: Error finding container 6a65478cafc38553b344e21dd70113ce277574e11bc0c0008569736640733c62: Status 404 returned error can't find the container with id 6a65478cafc38553b344e21dd70113ce277574e11bc0c0008569736640733c62 Nov 02 19:57:10.239057 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:10.239024 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" event={"ID":"0cd9fdd9-fd6a-415a-9d89-655b5637a192","Type":"ContainerStarted","Data":"6a65478cafc38553b344e21dd70113ce277574e11bc0c0008569736640733c62"} Nov 02 19:57:13.261848 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:13.261810 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" event={"ID":"0cd9fdd9-fd6a-415a-9d89-655b5637a192","Type":"ContainerStarted","Data":"e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80"} Nov 02 19:57:13.262256 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:13.262012 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:13.369237 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:13.369180 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" podStartSLOduration=1.739742474 podStartE2EDuration="4.369161294s" totalImagesPullingTime="2.62941882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:13.368397102 +0000 UTC m=+531.885169686" watchObservedRunningTime="2026-11-02 19:57:13.369161294 +0000 UTC m=+531.885933880" Nov 02 19:57:22.696578 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:57:22.696538 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:57:24.270508 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:24.270474 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 19:57:48.497298 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.497250 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs"] Nov 02 19:57:48.500909 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.500889 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.552240 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.552202 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.552512 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.552248 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rkw4\" (UniqueName: \"kubernetes.io/projected/649c1f83-2c79-4258-9822-4ed4628ad39e-kube-api-access-4rkw4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.552686 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.552660 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.552784 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.552711 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649c1f83-2c79-4258-9822-4ed4628ad39e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.552859 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.552843 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.552910 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.552872 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.553143 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.553104 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 19:57:48.598750 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.598718 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 19:57:48.598750 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.598742 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f7j5x\"" Nov 02 19:57:48.598963 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.598742 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 19:57:48.600666 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.600646 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs"] Nov 02 19:57:48.653776 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.653742 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.653776 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.653779 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.654020 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.653806 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.654020 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.653832 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rkw4\" (UniqueName: \"kubernetes.io/projected/649c1f83-2c79-4258-9822-4ed4628ad39e-kube-api-access-4rkw4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.654020 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.653886 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.654020 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.653908 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649c1f83-2c79-4258-9822-4ed4628ad39e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.654265 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.654243 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.654343 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.654260 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.654423 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.654397 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.656819 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.656795 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/649c1f83-2c79-4258-9822-4ed4628ad39e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.657107 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.657083 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649c1f83-2c79-4258-9822-4ed4628ad39e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.707398 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.707368 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rkw4\" (UniqueName: \"kubernetes.io/projected/649c1f83-2c79-4258-9822-4ed4628ad39e-kube-api-access-4rkw4\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-76pbs\" (UID: \"649c1f83-2c79-4258-9822-4ed4628ad39e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:48.814107 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:48.814065 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:49.177722 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:49.177693 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs"] Nov 02 19:57:49.180141 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:57:49.180108 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649c1f83_2c79_4258_9822_4ed4628ad39e.slice/crio-667d985dc97fc3b90e723208d426307daa34c9edfee9a3db530dc8e56cadd546 WatchSource:0}: Error finding container 667d985dc97fc3b90e723208d426307daa34c9edfee9a3db530dc8e56cadd546: Status 404 returned error can't find the container with id 667d985dc97fc3b90e723208d426307daa34c9edfee9a3db530dc8e56cadd546 Nov 02 19:57:49.441690 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:49.441604 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" event={"ID":"649c1f83-2c79-4258-9822-4ed4628ad39e","Type":"ContainerStarted","Data":"667d985dc97fc3b90e723208d426307daa34c9edfee9a3db530dc8e56cadd546"} Nov 02 19:57:51.454858 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:51.454822 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" event={"ID":"649c1f83-2c79-4258-9822-4ed4628ad39e","Type":"ContainerStarted","Data":"c7f7567122a92f4809ae77239c82319ecfcd4dce496300ccd9c491e961e9c067"} Nov 02 19:57:51.564106 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:51.564036 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" podStartSLOduration=1.987060343 podStartE2EDuration="3.564013812s" totalImagesPullingTime="1.576953469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:51.561028659 +0000 UTC m=+570.077801245" watchObservedRunningTime="2026-11-02 19:57:51.564013812 +0000 UTC m=+570.080786398" Nov 02 19:57:51.815015 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:51.814973 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:52.407536 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.407494 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk"] Nov 02 19:57:52.411273 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.411246 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.463426 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.463395 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk"] Nov 02 19:57:52.493052 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.493019 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.493216 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.493072 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.493216 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.493166 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2446135-0668-4948-99a4-cabed80f52cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.493379 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.493219 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6klb\" (UniqueName: \"kubernetes.io/projected/f2446135-0668-4948-99a4-cabed80f52cb-kube-api-access-l6klb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.493379 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.493239 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.493461 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.493407 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.507185 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.507149 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 19:57:52.594424 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.594387 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2446135-0668-4948-99a4-cabed80f52cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.594639 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.594447 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6klb\" (UniqueName: \"kubernetes.io/projected/f2446135-0668-4948-99a4-cabed80f52cb-kube-api-access-l6klb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.594639 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.594571 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.594757 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.594650 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.594757 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.594681 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.594842 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.594786 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.595066 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.595037 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.595184 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.595133 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.595184 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.595179 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.596789 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.596770 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2446135-0668-4948-99a4-cabed80f52cb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.596969 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.596953 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2446135-0668-4948-99a4-cabed80f52cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.647673 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.647641 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6klb\" (UniqueName: \"kubernetes.io/projected/f2446135-0668-4948-99a4-cabed80f52cb-kube-api-access-l6klb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-6mdnk\" (UID: \"f2446135-0668-4948-99a4-cabed80f52cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:52.694730 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:57:52.694648 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:57:52.724455 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:52.724421 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:53.088018 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:53.087994 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk"] Nov 02 19:57:53.089923 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:57:53.089898 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2446135_0668_4948_99a4_cabed80f52cb.slice/crio-6f8729ef4f1d173af5dee58b40273ea1afcb368930927edbb0ece189c873ee2b WatchSource:0}: Error finding container 6f8729ef4f1d173af5dee58b40273ea1afcb368930927edbb0ece189c873ee2b: Status 404 returned error can't find the container with id 6f8729ef4f1d173af5dee58b40273ea1afcb368930927edbb0ece189c873ee2b Nov 02 19:57:53.466683 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:53.466590 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" event={"ID":"f2446135-0668-4948-99a4-cabed80f52cb","Type":"ContainerStarted","Data":"6f8729ef4f1d173af5dee58b40273ea1afcb368930927edbb0ece189c873ee2b"} Nov 02 19:57:54.473825 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:54.473777 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" event={"ID":"f2446135-0668-4948-99a4-cabed80f52cb","Type":"ContainerStarted","Data":"d41ffc0731ab98a5680235ec23dcebc5c74dbb3cf85b5cf6c193ebbe4c7b47e9"} Nov 02 19:57:54.582124 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:54.582060 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" podStartSLOduration=1.9631060150000001 podStartE2EDuration="2.582045501s" totalImagesPullingTime="618.939486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:57:54.580757635 +0000 UTC m=+573.097530221" watchObservedRunningTime="2026-11-02 19:57:54.582045501 +0000 UTC m=+573.098818087" Nov 02 19:57:54.725059 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:54.724976 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:57:58.814352 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:58.814314 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:58.826819 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:58.826784 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:59.508946 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.508916 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-76pbs" Nov 02 19:57:59.865195 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.862058 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq"] Nov 02 19:57:59.867666 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.867633 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:57:59.917398 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.917365 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 19:57:59.918805 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.918781 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq"] Nov 02 19:57:59.961245 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.961208 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:57:59.961447 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.961346 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:57:59.961447 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.961419 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:57:59.961447 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.961443 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:57:59.961623 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.961544 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f51894d-a930-42e9-ab5a-a1727a3b5c71-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:57:59.961623 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:57:59.961570 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ll8p\" (UniqueName: \"kubernetes.io/projected/6f51894d-a930-42e9-ab5a-a1727a3b5c71-kube-api-access-5ll8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.062600 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.062562 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ll8p\" (UniqueName: \"kubernetes.io/projected/6f51894d-a930-42e9-ab5a-a1727a3b5c71-kube-api-access-5ll8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.062804 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.062622 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.062804 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.062681 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.062804 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.062746 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.062804 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.062782 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.063024 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.062821 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f51894d-a930-42e9-ab5a-a1727a3b5c71-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.063222 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.063195 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.063314 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.063247 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.063314 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.063271 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.064988 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.064969 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f51894d-a930-42e9-ab5a-a1727a3b5c71-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.065395 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.065379 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f51894d-a930-42e9-ab5a-a1727a3b5c71-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.124842 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.124765 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ll8p\" (UniqueName: \"kubernetes.io/projected/6f51894d-a930-42e9-ab5a-a1727a3b5c71-kube-api-access-5ll8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq\" (UID: \"6f51894d-a930-42e9-ab5a-a1727a3b5c71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.183324 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.183289 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:00.565500 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:00.565469 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq"] Nov 02 19:58:00.566715 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:58:00.566679 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f51894d_a930_42e9_ab5a_a1727a3b5c71.slice/crio-4cf245b9a75363f664c34d04c06ee215af71c01c26e1c1f0a3389ecdb82eb61b WatchSource:0}: Error finding container 4cf245b9a75363f664c34d04c06ee215af71c01c26e1c1f0a3389ecdb82eb61b: Status 404 returned error can't find the container with id 4cf245b9a75363f664c34d04c06ee215af71c01c26e1c1f0a3389ecdb82eb61b Nov 02 19:58:01.509705 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:01.509662 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" event={"ID":"6f51894d-a930-42e9-ab5a-a1727a3b5c71","Type":"ContainerStarted","Data":"50b402c71ee883f64a05503627d181d314df971faa46e2691128f6399b67bbe3"} Nov 02 19:58:01.510144 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:01.509723 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" event={"ID":"6f51894d-a930-42e9-ab5a-a1727a3b5c71","Type":"ContainerStarted","Data":"4cf245b9a75363f664c34d04c06ee215af71c01c26e1c1f0a3389ecdb82eb61b"} Nov 02 19:58:01.617672 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:01.617609 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" podStartSLOduration=1.989509942 podStartE2EDuration="2.617590985s" totalImagesPullingTime="628.081043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:01.616405662 +0000 UTC m=+580.133178247" watchObservedRunningTime="2026-11-02 19:58:01.617590985 +0000 UTC m=+580.134363571" Nov 02 19:58:02.184087 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:02.184052 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:02.725643 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:02.725603 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:58:02.738091 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:02.738063 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:58:03.532464 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:03.532435 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-6mdnk" Nov 02 19:58:07.100854 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.100805 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw"] Nov 02 19:58:07.105193 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.105164 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.156852 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.156806 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 19:58:07.202346 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.202313 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw"] Nov 02 19:58:07.233751 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.233715 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbd455b-6e9e-4c24-a9dd-3a029dede190-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.233751 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.233756 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.234025 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.233787 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.234025 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.233996 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5w2h\" (UniqueName: \"kubernetes.io/projected/8bbd455b-6e9e-4c24-a9dd-3a029dede190-kube-api-access-s5w2h\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.234143 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.234080 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.234143 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.234107 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.335197 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.335141 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbd455b-6e9e-4c24-a9dd-3a029dede190-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.335197 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.335198 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.335487 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.335225 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.335487 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.335320 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5w2h\" (UniqueName: \"kubernetes.io/projected/8bbd455b-6e9e-4c24-a9dd-3a029dede190-kube-api-access-s5w2h\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.335487 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.335377 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.335487 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.335401 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.336013 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.335976 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.336167 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.336115 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.336249 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.336226 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.338234 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.338203 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbd455b-6e9e-4c24-a9dd-3a029dede190-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.338234 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.338211 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbd455b-6e9e-4c24-a9dd-3a029dede190-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.388994 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.388911 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5w2h\" (UniqueName: \"kubernetes.io/projected/8bbd455b-6e9e-4c24-a9dd-3a029dede190-kube-api-access-s5w2h\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw\" (UID: \"8bbd455b-6e9e-4c24-a9dd-3a029dede190\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.428840 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.428803 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:07.817426 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:07.817401 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw"] Nov 02 19:58:07.819244 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:58:07.819216 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbd455b_6e9e_4c24_a9dd_3a029dede190.slice/crio-7b3c071f92b4c4e4b605d1b0dade0553ae3086d2cb21f27e03b1065b4e5bf1c6 WatchSource:0}: Error finding container 7b3c071f92b4c4e4b605d1b0dade0553ae3086d2cb21f27e03b1065b4e5bf1c6: Status 404 returned error can't find the container with id 7b3c071f92b4c4e4b605d1b0dade0553ae3086d2cb21f27e03b1065b4e5bf1c6 Nov 02 19:58:08.551297 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:08.551247 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" event={"ID":"8bbd455b-6e9e-4c24-a9dd-3a029dede190","Type":"ContainerStarted","Data":"d11301d0b099583a0992038ae8f267079a42345b2ef7d2395ae20391f44647b0"} Nov 02 19:58:08.551763 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:08.551357 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" event={"ID":"8bbd455b-6e9e-4c24-a9dd-3a029dede190","Type":"ContainerStarted","Data":"7b3c071f92b4c4e4b605d1b0dade0553ae3086d2cb21f27e03b1065b4e5bf1c6"} Nov 02 19:58:08.659330 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:08.659254 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" podStartSLOduration=1.052952808 podStartE2EDuration="1.65923986s" totalImagesPullingTime="606.287052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:08.656975479 +0000 UTC m=+587.173748064" watchObservedRunningTime="2026-11-02 19:58:08.65923986 +0000 UTC m=+587.176012473" Nov 02 19:58:09.428952 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:09.428919 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:10.184071 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.184042 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:10.196998 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.196972 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:10.571998 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.571969 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq" Nov 02 19:58:10.908133 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.908044 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6"] Nov 02 19:58:10.911961 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.911943 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:10.960916 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.960881 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 19:58:10.973237 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.973204 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6"] Nov 02 19:58:10.975490 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.975456 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:10.975622 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.975504 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwpl\" (UniqueName: \"kubernetes.io/projected/16c42059-424f-41a1-9108-ba08f2c79af9-kube-api-access-9dwpl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:10.975622 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.975599 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:10.975724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.975708 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c42059-424f-41a1-9108-ba08f2c79af9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:10.975795 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.975779 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:10.975848 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:10.975804 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077137 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077104 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077137 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077138 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwpl\" (UniqueName: \"kubernetes.io/projected/16c42059-424f-41a1-9108-ba08f2c79af9-kube-api-access-9dwpl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077465 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077160 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077465 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077324 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c42059-424f-41a1-9108-ba08f2c79af9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077465 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077425 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077465 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077456 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077667 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077587 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077667 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077629 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.077839 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.077819 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.079678 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.079649 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c42059-424f-41a1-9108-ba08f2c79af9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.079920 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.079903 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c42059-424f-41a1-9108-ba08f2c79af9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.130229 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.130191 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwpl\" (UniqueName: \"kubernetes.io/projected/16c42059-424f-41a1-9108-ba08f2c79af9-kube-api-access-9dwpl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6\" (UID: \"16c42059-424f-41a1-9108-ba08f2c79af9\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.222999 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.222918 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:11.590438 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:11.590412 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6"] Nov 02 19:58:11.591680 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:58:11.591652 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16c42059_424f_41a1_9108_ba08f2c79af9.slice/crio-a341facdfc880c3e46c02e7ec5f3942fc6e06db8ab5029754eb6b51629674f13 WatchSource:0}: Error finding container a341facdfc880c3e46c02e7ec5f3942fc6e06db8ab5029754eb6b51629674f13: Status 404 returned error can't find the container with id a341facdfc880c3e46c02e7ec5f3942fc6e06db8ab5029754eb6b51629674f13 Nov 02 19:58:12.573426 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:12.573392 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" event={"ID":"16c42059-424f-41a1-9108-ba08f2c79af9","Type":"ContainerStarted","Data":"a6c4917f9aafaeb9f6e9206fc122095639241bebfc565a98ab5ea96f70a21ee5"} Nov 02 19:58:12.573831 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:12.573433 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" event={"ID":"16c42059-424f-41a1-9108-ba08f2c79af9","Type":"ContainerStarted","Data":"a341facdfc880c3e46c02e7ec5f3942fc6e06db8ab5029754eb6b51629674f13"} Nov 02 19:58:12.680710 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:12.680657 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podStartSLOduration=2.105975774 podStartE2EDuration="2.680644024s" totalImagesPullingTime="574.66825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:12.678559491 +0000 UTC m=+591.195332076" watchObservedRunningTime="2026-11-02 19:58:12.680644024 +0000 UTC m=+591.197416608" Nov 02 19:58:13.223432 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:13.223387 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:17.429999 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:17.429955 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:17.442356 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:17.442330 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:17.613216 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:17.613185 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw" Nov 02 19:58:21.223347 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:21.223312 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:21.235917 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:21.235887 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:21.633928 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:21.633900 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 19:58:21.693501 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:21.693467 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:58:21.694176 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:21.694149 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 19:58:22.015867 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.015831 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd"] Nov 02 19:58:22.020820 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.020797 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.120066 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.120034 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd"] Nov 02 19:58:22.187411 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187377 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187411 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187414 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187618 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187441 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187618 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187593 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63815b6-e9af-4d31-8b8d-c309561f8433-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187690 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187623 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187690 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187642 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187758 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187746 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187793 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187775 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.187825 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.187798 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k7wj\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-kube-api-access-6k7wj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.288656 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288572 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63815b6-e9af-4d31-8b8d-c309561f8433-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.288656 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288616 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.288656 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288635 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288681 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288703 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288726 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7wj\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-kube-api-access-6k7wj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288776 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288800 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.288824 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.289081 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.289160 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289190 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.289187 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289667 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.289224 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.289667 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.289470 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63815b6-e9af-4d31-8b8d-c309561f8433-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.291163 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.291139 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.291386 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.291368 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.341928 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.341888 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.342059 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.341998 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7wj\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-kube-api-access-6k7wj\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-vxdqd\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.634081 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:22.633990 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:22.704983 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:58:22.704937 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:58:23.068340 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:23.068310 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd"] Nov 02 19:58:23.068753 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:58:23.068722 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63815b6_e9af_4d31_8b8d_c309561f8433.slice/crio-318fd2da68602dd8c07165427cde4702ca9f5d1b2971b4fcf7a12b791d4880cb WatchSource:0}: Error finding container 318fd2da68602dd8c07165427cde4702ca9f5d1b2971b4fcf7a12b791d4880cb: Status 404 returned error can't find the container with id 318fd2da68602dd8c07165427cde4702ca9f5d1b2971b4fcf7a12b791d4880cb Nov 02 19:58:23.071024 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:23.070992 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:58:23.071114 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:23.071077 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:58:23.071114 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:23.071107 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:58:23.654298 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:23.654251 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" event={"ID":"e63815b6-e9af-4d31-8b8d-c309561f8433","Type":"ContainerStarted","Data":"9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c"} Nov 02 19:58:23.654814 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:23.654338 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" event={"ID":"e63815b6-e9af-4d31-8b8d-c309561f8433","Type":"ContainerStarted","Data":"318fd2da68602dd8c07165427cde4702ca9f5d1b2971b4fcf7a12b791d4880cb"} Nov 02 19:58:23.766640 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:23.766586 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podStartSLOduration=2.766572892 podStartE2EDuration="2.766572892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:23.763792272 +0000 UTC m=+602.280564859" watchObservedRunningTime="2026-11-02 19:58:23.766572892 +0000 UTC m=+602.283345477" Nov 02 19:58:24.641905 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:24.634595 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:24.645408 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:24.645376 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 19:58:24.645629 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:24.645597 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 19:58:25.644307 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:25.641633 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 19:58:25.644307 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:25.641696 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 19:58:26.647307 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:26.646653 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 19:58:26.647307 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:26.646716 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 19:58:27.639701 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:27.639663 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 19:58:27.640003 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:27.639977 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 19:58:28.635647 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:28.635611 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 19:58:28.636267 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:28.636226 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 19:58:29.634572 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:29.634540 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 02 19:58:29.634753 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:29.634593 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 02 19:58:31.366576 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:31.366540 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:31.367330 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:31.366918 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:31.367921 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:31.367898 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 19:58:42.799673 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:42.799637 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f74f7c786-fr9rr"] Nov 02 19:58:42.803441 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:42.803420 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:42.855353 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:42.855321 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f74f7c786-fr9rr"] Nov 02 19:58:42.902351 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:42.902315 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44b1f0d7-2146-4520-bfc1-f7c56ddd37be-tls-cert\") pod \"authorino-6f74f7c786-fr9rr\" (UID: \"44b1f0d7-2146-4520-bfc1-f7c56ddd37be\") " pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:42.902506 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:42.902365 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2m6d\" (UniqueName: \"kubernetes.io/projected/44b1f0d7-2146-4520-bfc1-f7c56ddd37be-kube-api-access-v2m6d\") pod \"authorino-6f74f7c786-fr9rr\" (UID: \"44b1f0d7-2146-4520-bfc1-f7c56ddd37be\") " pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:43.003122 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.003083 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44b1f0d7-2146-4520-bfc1-f7c56ddd37be-tls-cert\") pod \"authorino-6f74f7c786-fr9rr\" (UID: \"44b1f0d7-2146-4520-bfc1-f7c56ddd37be\") " pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:43.003330 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.003143 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2m6d\" (UniqueName: \"kubernetes.io/projected/44b1f0d7-2146-4520-bfc1-f7c56ddd37be-kube-api-access-v2m6d\") pod \"authorino-6f74f7c786-fr9rr\" (UID: \"44b1f0d7-2146-4520-bfc1-f7c56ddd37be\") " pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:43.005799 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.005772 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44b1f0d7-2146-4520-bfc1-f7c56ddd37be-tls-cert\") pod \"authorino-6f74f7c786-fr9rr\" (UID: \"44b1f0d7-2146-4520-bfc1-f7c56ddd37be\") " pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:43.056294 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.056208 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2m6d\" (UniqueName: \"kubernetes.io/projected/44b1f0d7-2146-4520-bfc1-f7c56ddd37be-kube-api-access-v2m6d\") pod \"authorino-6f74f7c786-fr9rr\" (UID: \"44b1f0d7-2146-4520-bfc1-f7c56ddd37be\") " pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:43.114896 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.114864 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f74f7c786-fr9rr" Nov 02 19:58:43.476041 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.476014 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f74f7c786-fr9rr"] Nov 02 19:58:43.478309 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:58:43.478261 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b1f0d7_2146_4520_bfc1_f7c56ddd37be.slice/crio-1c74318479238a756dd83c9bcdf9817f6e416891d5c96e890216050ac3547b63 WatchSource:0}: Error finding container 1c74318479238a756dd83c9bcdf9817f6e416891d5c96e890216050ac3547b63: Status 404 returned error can't find the container with id 1c74318479238a756dd83c9bcdf9817f6e416891d5c96e890216050ac3547b63 Nov 02 19:58:43.479626 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.479607 5800 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:58:43.759658 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:43.759626 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f74f7c786-fr9rr" event={"ID":"44b1f0d7-2146-4520-bfc1-f7c56ddd37be","Type":"ContainerStarted","Data":"1c74318479238a756dd83c9bcdf9817f6e416891d5c96e890216050ac3547b63"} Nov 02 19:58:45.772685 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:45.772629 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f74f7c786-fr9rr" event={"ID":"44b1f0d7-2146-4520-bfc1-f7c56ddd37be","Type":"ContainerStarted","Data":"f4e4a8a3746e940dde8f33b1ac134a55dbb3ab006d6f02df98219939082a410a"} Nov 02 19:58:45.924961 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:45.924894 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f74f7c786-fr9rr" podStartSLOduration=2.645569939 podStartE2EDuration="3.924872408s" totalImagesPullingTime="1.279302469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:58:45.879163505 +0000 UTC m=+624.395936091" watchObservedRunningTime="2026-11-02 19:58:45.924872408 +0000 UTC m=+624.441644996" Nov 02 19:58:45.928161 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:45.928080 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d9d749ccc-h4fh9"] Nov 02 19:58:45.928560 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:45.928529 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" podUID="567e8076-1152-4132-8a42-3e3959dbdca2" containerName="authorino" containerID="cri-o://cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33" gracePeriod=30 Nov 02 19:58:46.213625 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.213592 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:58:46.241678 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.241644 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/567e8076-1152-4132-8a42-3e3959dbdca2-tls-cert\" (UniqueName: \"kubernetes.io/secret/567e8076-1152-4132-8a42-3e3959dbdca2-tls-cert\") pod \"567e8076-1152-4132-8a42-3e3959dbdca2\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " Nov 02 19:58:46.241848 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.241708 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567e8076-1152-4132-8a42-3e3959dbdca2-kube-api-access-6tvb4\" (UniqueName: \"kubernetes.io/projected/567e8076-1152-4132-8a42-3e3959dbdca2-kube-api-access-6tvb4\") pod \"567e8076-1152-4132-8a42-3e3959dbdca2\" (UID: \"567e8076-1152-4132-8a42-3e3959dbdca2\") " Nov 02 19:58:46.243810 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.243779 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567e8076-1152-4132-8a42-3e3959dbdca2-kube-api-access-6tvb4" pod "567e8076-1152-4132-8a42-3e3959dbdca2" (UID: "567e8076-1152-4132-8a42-3e3959dbdca2"). InnerVolumeSpecName "kube-api-access-6tvb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:58:46.254956 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.254928 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567e8076-1152-4132-8a42-3e3959dbdca2-tls-cert" pod "567e8076-1152-4132-8a42-3e3959dbdca2" (UID: "567e8076-1152-4132-8a42-3e3959dbdca2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:58:46.343468 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.343376 5800 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/567e8076-1152-4132-8a42-3e3959dbdca2-tls-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:58:46.343468 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.343413 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tvb4\" (UniqueName: \"kubernetes.io/projected/567e8076-1152-4132-8a42-3e3959dbdca2-kube-api-access-6tvb4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 19:58:46.779637 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.779600 5800 generic.go:356] "Generic (PLEG): container finished" podID="567e8076-1152-4132-8a42-3e3959dbdca2" containerID="cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33" exitCode=0 Nov 02 19:58:46.780092 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.779654 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" Nov 02 19:58:46.780092 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.779682 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" event={"ID":"567e8076-1152-4132-8a42-3e3959dbdca2","Type":"ContainerDied","Data":"cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33"} Nov 02 19:58:46.780092 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.779725 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d9d749ccc-h4fh9" event={"ID":"567e8076-1152-4132-8a42-3e3959dbdca2","Type":"ContainerDied","Data":"1c8c445fc7ea6d981398582c8a5c28d9cbcb7684d4fc2d5c9744436418a30b45"} Nov 02 19:58:46.780092 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.779746 5800 scope.go:122] "RemoveContainer" containerID="cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33" Nov 02 19:58:46.790022 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.789768 5800 scope.go:122] "RemoveContainer" containerID="cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33" Nov 02 19:58:46.790185 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:58:46.790096 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33\": container with ID starting with cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33 not found: ID does not exist" containerID="cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33" Nov 02 19:58:46.790185 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.790128 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33"} err="failed to get container status \"cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33\": rpc error: code = NotFound desc = could not find container \"cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33\": container with ID starting with cfec558c53bf310c08970e7e0f75b6a94e06221ffdb212674335381c9d2ddf33 not found: ID does not exist" Nov 02 19:58:46.932315 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.932260 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d9d749ccc-h4fh9"] Nov 02 19:58:46.978274 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:46.978243 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d9d749ccc-h4fh9"] Nov 02 19:58:47.621863 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:58:47.621828 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567e8076-1152-4132-8a42-3e3959dbdca2" path="/var/lib/kubelet/pods/567e8076-1152-4132-8a42-3e3959dbdca2/volumes" Nov 02 19:58:52.696412 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:58:52.696366 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:59:14.058771 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.058688 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r"] Nov 02 19:59:14.059238 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.059121 5800 cpu_manager.go:414] "containerMap: removing container" podUID="567e8076-1152-4132-8a42-3e3959dbdca2" containerName="authorino" Nov 02 19:59:14.059238 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.059133 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567e8076-1152-4132-8a42-3e3959dbdca2" containerName="authorino" Nov 02 19:59:14.059238 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.059202 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="567e8076-1152-4132-8a42-3e3959dbdca2" containerName="authorino" Nov 02 19:59:14.062473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.062446 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.123449 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.123403 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4db200eb-openshift-default-dockercfg-qhcw7\"" Nov 02 19:59:14.159518 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.159487 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r"] Nov 02 19:59:14.227337 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227273 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c5749c-2a39-446d-a037-71501cd76e10-istiod-ca-cert\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227534 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227391 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-envoy\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227534 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227475 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9z5\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-kube-api-access-nf9z5\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227534 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227503 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-credential-socket\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227534 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227527 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-certs\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227688 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227558 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-socket\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227688 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227591 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-data\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227688 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227649 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-istio-token\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.227688 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.227680 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c5749c-2a39-446d-a037-71501cd76e10-istio-podinfo\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329073 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.328959 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c5749c-2a39-446d-a037-71501cd76e10-istiod-ca-cert\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329073 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329043 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-envoy\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329324 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329085 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9z5\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-kube-api-access-nf9z5\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329324 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329112 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-credential-socket\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329324 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329141 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-certs\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329324 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329176 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-socket\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329324 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329203 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-data\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329673 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329511 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-istio-token\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329673 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329564 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-socket\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329673 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329594 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c5749c-2a39-446d-a037-71501cd76e10-istio-podinfo\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329829 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329672 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-certs\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329829 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329755 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-credential-socket\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.329829 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.329799 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c5749c-2a39-446d-a037-71501cd76e10-istiod-ca-cert\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.330084 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.330065 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-data\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.332148 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.332127 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-envoy\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.332276 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.332188 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c5749c-2a39-446d-a037-71501cd76e10-istio-podinfo\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.382431 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.382397 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9z5\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-kube-api-access-nf9z5\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.382564 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.382526 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-istio-token\") pod \"e2e-ait-4db200eb-openshift-default-7457499b79-gk52r\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:14.675476 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:14.675397 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:15.097174 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:15.097144 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r"] Nov 02 19:59:15.099796 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:15.099768 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c5749c_2a39_446d_a037_71501cd76e10.slice/crio-8606919c4fd7a8011797b21abceef1faaa16e79a62a5a7a686c55b83cad6fe54 WatchSource:0}: Error finding container 8606919c4fd7a8011797b21abceef1faaa16e79a62a5a7a686c55b83cad6fe54: Status 404 returned error can't find the container with id 8606919c4fd7a8011797b21abceef1faaa16e79a62a5a7a686c55b83cad6fe54 Nov 02 19:59:15.102219 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:15.102184 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:15.102373 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:15.102262 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:15.102373 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:15.102324 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:15.940610 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:15.940571 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" event={"ID":"38c5749c-2a39-446d-a037-71501cd76e10","Type":"ContainerStarted","Data":"21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1"} Nov 02 19:59:15.940783 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:15.940638 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" event={"ID":"38c5749c-2a39-446d-a037-71501cd76e10","Type":"ContainerStarted","Data":"8606919c4fd7a8011797b21abceef1faaa16e79a62a5a7a686c55b83cad6fe54"} Nov 02 19:59:16.099323 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.099254 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" podStartSLOduration=2.099239857 podStartE2EDuration="2.099239857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:16.051553816 +0000 UTC m=+654.568326402" watchObservedRunningTime="2026-11-02 19:59:16.099239857 +0000 UTC m=+654.616012421" Nov 02 19:59:16.254882 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.254851 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92"] Nov 02 19:59:16.258659 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.258631 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.308149 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.308117 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-70ebea5b-openshift-default-dockercfg-mfnrq\"" Nov 02 19:59:16.320702 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.320667 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92"] Nov 02 19:59:16.347537 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347498 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-data\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.347537 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347540 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba07593-5b6f-4158-9a9a-8c299080b548-istiod-ca-cert\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.347772 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347597 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-certs\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.347772 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347615 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-envoy\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.347772 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347641 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-credential-socket\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.347772 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347662 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-socket\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.347772 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347745 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-podinfo\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.348024 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347801 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-token\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.348024 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.347830 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gscb5\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-kube-api-access-gscb5\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.448469 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.448422 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-credential-socket\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.448652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.448478 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-socket\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.448652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.448508 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-podinfo\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.448652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.448540 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-token\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.448652 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.448581 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gscb5\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-kube-api-access-gscb5\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.448912 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.448678 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-data\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.448912 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.448712 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba07593-5b6f-4158-9a9a-8c299080b548-istiod-ca-cert\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.449085 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.449057 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-credential-socket\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.449155 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.449083 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-socket\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.449234 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.449218 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-certs\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.449272 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.449245 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-envoy\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.449545 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.449522 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-certs\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.449709 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.449687 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba07593-5b6f-4158-9a9a-8c299080b548-istiod-ca-cert\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.449840 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.449820 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-data\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.451269 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.451250 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-podinfo\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.451454 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.451437 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-envoy\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.505103 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.505023 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gscb5\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-kube-api-access-gscb5\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.506791 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.506761 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-token\") pod \"e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.576140 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.576109 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:16.675717 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.675685 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:16.681166 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.681140 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:16.947177 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.947057 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:16.947978 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:16.947954 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:17.001464 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:17.001434 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92"] Nov 02 19:59:17.003107 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:17.003081 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba07593_5b6f_4158_9a9a_8c299080b548.slice/crio-d44f331d74d9dd5e5d0f2b431f9070cfbb3a2c661b88f1a29686c50a150e20da WatchSource:0}: Error finding container d44f331d74d9dd5e5d0f2b431f9070cfbb3a2c661b88f1a29686c50a150e20da: Status 404 returned error can't find the container with id d44f331d74d9dd5e5d0f2b431f9070cfbb3a2c661b88f1a29686c50a150e20da Nov 02 19:59:17.005380 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:17.005350 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:17.005488 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:17.005428 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:17.005488 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:17.005471 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:17.954063 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:17.954025 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" event={"ID":"1ba07593-5b6f-4158-9a9a-8c299080b548","Type":"ContainerStarted","Data":"249bed269a32c21e290d3d63f66c7a0a3f2f6b1beb33788405c5bec0ddb8ab29"} Nov 02 19:59:17.954452 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:17.954082 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" event={"ID":"1ba07593-5b6f-4158-9a9a-8c299080b548","Type":"ContainerStarted","Data":"d44f331d74d9dd5e5d0f2b431f9070cfbb3a2c661b88f1a29686c50a150e20da"} Nov 02 19:59:18.064692 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:18.064637 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podStartSLOduration=2.064623999 podStartE2EDuration="2.064623999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:18.062957601 +0000 UTC m=+656.579730189" watchObservedRunningTime="2026-11-02 19:59:18.064623999 +0000 UTC m=+656.581396581" Nov 02 19:59:18.577189 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:18.577154 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:18.583340 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:18.583311 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:18.960126 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:18.960031 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:18.961084 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:18.961063 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:21.969951 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:21.969872 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2"] Nov 02 19:59:21.974983 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:21.974952 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.007373 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.007331 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8bv\" (UniqueName: \"kubernetes.io/projected/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-kube-api-access-jq8bv\") pod \"payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.007557 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.007518 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.070350 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.070316 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a0d19e8c\"" Nov 02 19:59:22.070533 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.070434 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a0d19e8c-dockercfg-xvwsq\"" Nov 02 19:59:22.072873 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.072850 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2"] Nov 02 19:59:22.108747 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.108712 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.108903 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.108781 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8bv\" (UniqueName: \"kubernetes.io/projected/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-kube-api-access-jq8bv\") pod \"payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.109473 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.109455 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.124308 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.124252 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb"] Nov 02 19:59:22.128028 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.128007 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.163732 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.163701 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8bv\" (UniqueName: \"kubernetes.io/projected/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-kube-api-access-jq8bv\") pod \"payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.180005 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.179977 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb"] Nov 02 19:59:22.209865 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.209832 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.210038 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.209895 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mmg\" (UniqueName: \"kubernetes.io/projected/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-kube-api-access-x9mmg\") pod \"payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.287939 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.287895 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.311650 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.311617 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.311794 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.311717 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mmg\" (UniqueName: \"kubernetes.io/projected/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-kube-api-access-x9mmg\") pod \"payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.312760 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.312732 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.365669 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.365630 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mmg\" (UniqueName: \"kubernetes.io/projected/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-kube-api-access-x9mmg\") pod \"payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.441270 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.441235 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:22.654908 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.654880 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2"] Nov 02 19:59:22.656149 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:22.656114 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b39ce6_59a3_4e67_9a40_4ea6f6b56e4a.slice/crio-5737ffd9f0d3a90578acb0214fdda31e7ed4a3e813b044605816b2443905ada8 WatchSource:0}: Error finding container 5737ffd9f0d3a90578acb0214fdda31e7ed4a3e813b044605816b2443905ada8: Status 404 returned error can't find the container with id 5737ffd9f0d3a90578acb0214fdda31e7ed4a3e813b044605816b2443905ada8 Nov 02 19:59:22.696918 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:59:22.696883 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:59:22.806842 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.806814 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb"] Nov 02 19:59:22.808160 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:22.808132 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec33fbe_1eab_4db3_b85b_e1abb7a464dd.slice/crio-de65a343af8f78ba07bc831d06f0f0099c47d6409c3dd25ccb421582f94d268d WatchSource:0}: Error finding container de65a343af8f78ba07bc831d06f0f0099c47d6409c3dd25ccb421582f94d268d: Status 404 returned error can't find the container with id de65a343af8f78ba07bc831d06f0f0099c47d6409c3dd25ccb421582f94d268d Nov 02 19:59:22.983412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.983309 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" event={"ID":"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a","Type":"ContainerStarted","Data":"fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08"} Nov 02 19:59:22.983412 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.983382 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" event={"ID":"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a","Type":"ContainerStarted","Data":"5737ffd9f0d3a90578acb0214fdda31e7ed4a3e813b044605816b2443905ada8"} Nov 02 19:59:22.983882 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.983416 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:22.984969 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.984945 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" event={"ID":"dec33fbe-1eab-4db3-b85b-e1abb7a464dd","Type":"ContainerStarted","Data":"f3369ce41a2b6a01776708139f9c5b69f977e692a82bf46bf635612d793e67ce"} Nov 02 19:59:22.985070 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.984988 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" event={"ID":"dec33fbe-1eab-4db3-b85b-e1abb7a464dd","Type":"ContainerStarted","Data":"de65a343af8f78ba07bc831d06f0f0099c47d6409c3dd25ccb421582f94d268d"} Nov 02 19:59:22.985070 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:22.985022 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:23.091337 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:23.091268 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" podStartSLOduration=2.0912528950000002 podStartE2EDuration="2.091252895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:23.088433015 +0000 UTC m=+661.605205600" watchObservedRunningTime="2026-11-02 19:59:23.091252895 +0000 UTC m=+661.608025480" Nov 02 19:59:23.195220 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:23.195167 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" podStartSLOduration=1.195152135 podStartE2EDuration="1.195152135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:23.193375037 +0000 UTC m=+661.710147625" watchObservedRunningTime="2026-11-02 19:59:23.195152135 +0000 UTC m=+661.711924713" Nov 02 19:59:26.519716 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.519684 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49"] Nov 02 19:59:26.523579 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.523558 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.556817 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.556789 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d780c0-9801-4f46-bd90-e0f102a877d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.556980 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.556842 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgqq\" (UniqueName: \"kubernetes.io/projected/82d780c0-9801-4f46-bd90-e0f102a877d7-kube-api-access-6zgqq\") pod \"payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.620206 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.620173 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3acf2229\"" Nov 02 19:59:26.620206 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.620193 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3acf2229-dockercfg-twxvl\"" Nov 02 19:59:26.622661 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.622633 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49"] Nov 02 19:59:26.658307 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.658252 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d780c0-9801-4f46-bd90-e0f102a877d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.658481 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.658342 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgqq\" (UniqueName: \"kubernetes.io/projected/82d780c0-9801-4f46-bd90-e0f102a877d7-kube-api-access-6zgqq\") pod \"payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.659130 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.659101 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d780c0-9801-4f46-bd90-e0f102a877d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.674530 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.674501 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr"] Nov 02 19:59:26.678621 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.678603 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:26.711010 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.710980 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgqq\" (UniqueName: \"kubernetes.io/projected/82d780c0-9801-4f46-bd90-e0f102a877d7-kube-api-access-6zgqq\") pod \"payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.731094 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.731066 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr"] Nov 02 19:59:26.759610 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.759582 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmtf\" (UniqueName: \"kubernetes.io/projected/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-kube-api-access-6gmtf\") pod \"payload-processing-e2e-ait-3acf2229-b6f78675-f4srr\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:26.759742 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.759633 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-plugins-config-volume\") pod \"payload-processing-e2e-ait-3acf2229-b6f78675-f4srr\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:26.837720 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.837627 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:26.860681 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.860651 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-plugins-config-volume\") pod \"payload-processing-e2e-ait-3acf2229-b6f78675-f4srr\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:26.860825 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.860807 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmtf\" (UniqueName: \"kubernetes.io/projected/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-kube-api-access-6gmtf\") pod \"payload-processing-e2e-ait-3acf2229-b6f78675-f4srr\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:26.861751 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.861730 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-plugins-config-volume\") pod \"payload-processing-e2e-ait-3acf2229-b6f78675-f4srr\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:26.913831 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.913782 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmtf\" (UniqueName: \"kubernetes.io/projected/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-kube-api-access-6gmtf\") pod \"payload-processing-e2e-ait-3acf2229-b6f78675-f4srr\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:26.991624 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:26.991574 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:27.201987 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:27.201892 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49"] Nov 02 19:59:27.205169 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:27.205133 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d780c0_9801_4f46_bd90_e0f102a877d7.slice/crio-3ca39faac4d8c6ccf966e947f335cd88babda5a14ab82e6e2f22dc527f0e8840 WatchSource:0}: Error finding container 3ca39faac4d8c6ccf966e947f335cd88babda5a14ab82e6e2f22dc527f0e8840: Status 404 returned error can't find the container with id 3ca39faac4d8c6ccf966e947f335cd88babda5a14ab82e6e2f22dc527f0e8840 Nov 02 19:59:27.355417 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:27.355389 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr"] Nov 02 19:59:27.357218 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:27.357175 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8cbaad_4073_4be8_8ff9_5551b6eca79f.slice/crio-8dcb404e566c8c84607033282e8d3995a8b29582243c7bf93926b2a59247eeae WatchSource:0}: Error finding container 8dcb404e566c8c84607033282e8d3995a8b29582243c7bf93926b2a59247eeae: Status 404 returned error can't find the container with id 8dcb404e566c8c84607033282e8d3995a8b29582243c7bf93926b2a59247eeae Nov 02 19:59:28.016274 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.016227 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" event={"ID":"82d780c0-9801-4f46-bd90-e0f102a877d7","Type":"ContainerStarted","Data":"ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133"} Nov 02 19:59:28.016737 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.016319 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" event={"ID":"82d780c0-9801-4f46-bd90-e0f102a877d7","Type":"ContainerStarted","Data":"3ca39faac4d8c6ccf966e947f335cd88babda5a14ab82e6e2f22dc527f0e8840"} Nov 02 19:59:28.016737 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.016356 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:28.017751 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.017727 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" event={"ID":"0b8cbaad-4073-4be8-8ff9-5551b6eca79f","Type":"ContainerStarted","Data":"0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c"} Nov 02 19:59:28.017861 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.017780 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:28.017861 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.017805 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" event={"ID":"0b8cbaad-4073-4be8-8ff9-5551b6eca79f","Type":"ContainerStarted","Data":"8dcb404e566c8c84607033282e8d3995a8b29582243c7bf93926b2a59247eeae"} Nov 02 19:59:28.123768 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.123712 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" podStartSLOduration=2.123696861 podStartE2EDuration="2.123696861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:28.122159512 +0000 UTC m=+666.638932098" watchObservedRunningTime="2026-11-02 19:59:28.123696861 +0000 UTC m=+666.640469446" Nov 02 19:59:28.230691 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:28.230623 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" podStartSLOduration=2.230607488 podStartE2EDuration="2.230607488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:28.227561627 +0000 UTC m=+666.744334213" watchObservedRunningTime="2026-11-02 19:59:28.230607488 +0000 UTC m=+666.747380104" Nov 02 19:59:29.026444 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.026411 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2"] Nov 02 19:59:29.030256 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.030235 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.082719 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.082667 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4db200eb\"" Nov 02 19:59:29.082948 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.082778 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a398715-373f-4f62-83aa-e1629a53806f-plugins-config-volume\") pod \"payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.082948 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.082820 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4db200eb-dockercfg-d8rfb\"" Nov 02 19:59:29.083101 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.082997 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmj5\" (UniqueName: \"kubernetes.io/projected/8a398715-373f-4f62-83aa-e1629a53806f-kube-api-access-jdmj5\") pod \"payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.131739 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.131708 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2"] Nov 02 19:59:29.184177 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.184144 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a398715-373f-4f62-83aa-e1629a53806f-plugins-config-volume\") pod \"payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.184392 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.184200 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmj5\" (UniqueName: \"kubernetes.io/projected/8a398715-373f-4f62-83aa-e1629a53806f-kube-api-access-jdmj5\") pod \"payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.185066 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.185043 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a398715-373f-4f62-83aa-e1629a53806f-plugins-config-volume\") pod \"payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.238433 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.238401 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmj5\" (UniqueName: \"kubernetes.io/projected/8a398715-373f-4f62-83aa-e1629a53806f-kube-api-access-jdmj5\") pod \"payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.344988 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.344905 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:29.713082 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:29.713056 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2"] Nov 02 19:59:29.715005 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:29.714979 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a398715_373f_4f62_83aa_e1629a53806f.slice/crio-8de10cb0ba58fcacac311efb69fec82326bbaf605bfc380a9e6b9fe09a292c79 WatchSource:0}: Error finding container 8de10cb0ba58fcacac311efb69fec82326bbaf605bfc380a9e6b9fe09a292c79: Status 404 returned error can't find the container with id 8de10cb0ba58fcacac311efb69fec82326bbaf605bfc380a9e6b9fe09a292c79 Nov 02 19:59:30.031790 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:30.031751 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" event={"ID":"8a398715-373f-4f62-83aa-e1629a53806f","Type":"ContainerStarted","Data":"2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5"} Nov 02 19:59:30.032315 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:30.031818 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:30.032315 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:30.031840 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" event={"ID":"8a398715-373f-4f62-83aa-e1629a53806f","Type":"ContainerStarted","Data":"8de10cb0ba58fcacac311efb69fec82326bbaf605bfc380a9e6b9fe09a292c79"} Nov 02 19:59:30.140871 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:30.140819 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" podStartSLOduration=2.1408042529999998 podStartE2EDuration="2.140804253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:30.138782096 +0000 UTC m=+668.655554682" watchObservedRunningTime="2026-11-02 19:59:30.140804253 +0000 UTC m=+668.657576838" Nov 02 19:59:31.290694 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.290655 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw"] Nov 02 19:59:31.294608 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.294591 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.348136 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.348104 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw"] Nov 02 19:59:31.391981 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.391860 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-70ebea5b-dockercfg-d6s7t\"" Nov 02 19:59:31.391981 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.391924 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-70ebea5b\"" Nov 02 19:59:31.410180 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.410147 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzncl\" (UniqueName: \"kubernetes.io/projected/0681b630-2424-48af-98df-b04c631a60b8-kube-api-access-lzncl\") pod \"payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.410359 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.410189 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0681b630-2424-48af-98df-b04c631a60b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.449620 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.449585 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf"] Nov 02 19:59:31.453743 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.453719 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.509335 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.509272 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf"] Nov 02 19:59:31.510971 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.510938 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzncl\" (UniqueName: \"kubernetes.io/projected/0681b630-2424-48af-98df-b04c631a60b8-kube-api-access-lzncl\") pod \"payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.511163 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.510997 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0681b630-2424-48af-98df-b04c631a60b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.511163 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.511072 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szprp\" (UniqueName: \"kubernetes.io/projected/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-kube-api-access-szprp\") pod \"payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.511163 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.511131 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-plugins-config-volume\") pod \"payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.511770 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.511747 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0681b630-2424-48af-98df-b04c631a60b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.564485 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.564409 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzncl\" (UniqueName: \"kubernetes.io/projected/0681b630-2424-48af-98df-b04c631a60b8-kube-api-access-lzncl\") pod \"payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.607758 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.607730 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:31.612237 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.612203 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szprp\" (UniqueName: \"kubernetes.io/projected/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-kube-api-access-szprp\") pod \"payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.612396 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.612268 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-plugins-config-volume\") pod \"payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.613098 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.613076 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-plugins-config-volume\") pod \"payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.665668 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.665588 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szprp\" (UniqueName: \"kubernetes.io/projected/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-kube-api-access-szprp\") pod \"payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.766837 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.766799 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:31.982007 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:31.981982 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw"] Nov 02 19:59:31.983649 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:31.983620 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0681b630_2424_48af_98df_b04c631a60b8.slice/crio-7a61212041b1783b3a66ff487520bd97e2de94d66272ae36f51cc61faa87556e WatchSource:0}: Error finding container 7a61212041b1783b3a66ff487520bd97e2de94d66272ae36f51cc61faa87556e: Status 404 returned error can't find the container with id 7a61212041b1783b3a66ff487520bd97e2de94d66272ae36f51cc61faa87556e Nov 02 19:59:32.046326 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:32.046292 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" event={"ID":"0681b630-2424-48af-98df-b04c631a60b8","Type":"ContainerStarted","Data":"7a61212041b1783b3a66ff487520bd97e2de94d66272ae36f51cc61faa87556e"} Nov 02 19:59:32.138098 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:32.138061 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4990dfd9_5aa3_4154_b130_fb9ac1bd6c42.slice/crio-e0edafd4d6854b8d241c35127f448915b752ae6d5a9f04c1858824d3455f24bf WatchSource:0}: Error finding container e0edafd4d6854b8d241c35127f448915b752ae6d5a9f04c1858824d3455f24bf: Status 404 returned error can't find the container with id e0edafd4d6854b8d241c35127f448915b752ae6d5a9f04c1858824d3455f24bf Nov 02 19:59:32.139161 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:32.139124 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf"] Nov 02 19:59:33.054811 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.054770 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" event={"ID":"0681b630-2424-48af-98df-b04c631a60b8","Type":"ContainerStarted","Data":"017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f"} Nov 02 19:59:33.055224 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.054870 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:33.063661 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.063624 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" event={"ID":"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42","Type":"ContainerStarted","Data":"fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d"} Nov 02 19:59:33.063780 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.063703 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" event={"ID":"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42","Type":"ContainerStarted","Data":"e0edafd4d6854b8d241c35127f448915b752ae6d5a9f04c1858824d3455f24bf"} Nov 02 19:59:33.063815 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.063782 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:33.160608 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.160546 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" podStartSLOduration=2.1605295 podStartE2EDuration="2.1605295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:33.15983349 +0000 UTC m=+671.676606076" watchObservedRunningTime="2026-11-02 19:59:33.1605295 +0000 UTC m=+671.677302086" Nov 02 19:59:33.267411 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.267352 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" podStartSLOduration=2.267339953 podStartE2EDuration="2.267339953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:33.265854294 +0000 UTC m=+671.782626880" watchObservedRunningTime="2026-11-02 19:59:33.267339953 +0000 UTC m=+671.784112538" Nov 02 19:59:33.994477 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.994449 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 19:59:33.994661 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:33.994519 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 19:59:39.026504 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:39.026474 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 19:59:39.026923 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:39.026550 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 19:59:40.182032 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.181994 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv"] Nov 02 19:59:40.186521 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.186493 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.239783 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.239739 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv"] Nov 02 19:59:40.281468 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.281431 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-172adb07-dockercfg-5qgsf\"" Nov 02 19:59:40.281647 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.281502 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-172adb07\"" Nov 02 19:59:40.330752 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.330717 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a31be42-1cbb-4599-bc04-4529d37f597a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.330937 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.330811 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzl2g\" (UniqueName: \"kubernetes.io/projected/2a31be42-1cbb-4599-bc04-4529d37f597a-kube-api-access-kzl2g\") pod \"payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.333489 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.333465 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp"] Nov 02 19:59:40.337176 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.337161 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.389867 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.389824 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp"] Nov 02 19:59:40.431936 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.431902 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjjg\" (UniqueName: \"kubernetes.io/projected/a32704be-d93d-48c0-b184-1c63d92b8b28-kube-api-access-mzjjg\") pod \"payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.431936 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.431945 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl2g\" (UniqueName: \"kubernetes.io/projected/2a31be42-1cbb-4599-bc04-4529d37f597a-kube-api-access-kzl2g\") pod \"payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.432188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.432115 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a31be42-1cbb-4599-bc04-4529d37f597a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.432188 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.432162 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32704be-d93d-48c0-b184-1c63d92b8b28-plugins-config-volume\") pod \"payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.432890 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.432870 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a31be42-1cbb-4599-bc04-4529d37f597a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.486025 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.485987 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzl2g\" (UniqueName: \"kubernetes.io/projected/2a31be42-1cbb-4599-bc04-4529d37f597a-kube-api-access-kzl2g\") pod \"payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.502198 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.502168 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:40.533239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.533201 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32704be-d93d-48c0-b184-1c63d92b8b28-plugins-config-volume\") pod \"payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.533425 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.533264 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjjg\" (UniqueName: \"kubernetes.io/projected/a32704be-d93d-48c0-b184-1c63d92b8b28-kube-api-access-mzjjg\") pod \"payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.534586 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.534554 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32704be-d93d-48c0-b184-1c63d92b8b28-plugins-config-volume\") pod \"payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.591197 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.590630 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjjg\" (UniqueName: \"kubernetes.io/projected/a32704be-d93d-48c0-b184-1c63d92b8b28-kube-api-access-mzjjg\") pod \"payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.651432 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.651391 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:40.886029 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:40.886001 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv"] Nov 02 19:59:40.887482 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:40.887454 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a31be42_1cbb_4599_bc04_4529d37f597a.slice/crio-0972170cd62697c4d3a6454bd252baef5389b67c8cde708516387202f198a0d7 WatchSource:0}: Error finding container 0972170cd62697c4d3a6454bd252baef5389b67c8cde708516387202f198a0d7: Status 404 returned error can't find the container with id 0972170cd62697c4d3a6454bd252baef5389b67c8cde708516387202f198a0d7 Nov 02 19:59:41.023239 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.023210 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp"] Nov 02 19:59:41.025277 ip-10-0-134-13 kubenswrapper[5800]: W1102 19:59:41.025250 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32704be_d93d_48c0_b184_1c63d92b8b28.slice/crio-dd0ac07574150c1ea8bc34bdc319a5b120bf5b674eb7753a728b9736ba78c1ef WatchSource:0}: Error finding container dd0ac07574150c1ea8bc34bdc319a5b120bf5b674eb7753a728b9736ba78c1ef: Status 404 returned error can't find the container with id dd0ac07574150c1ea8bc34bdc319a5b120bf5b674eb7753a728b9736ba78c1ef Nov 02 19:59:41.040821 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.040795 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 19:59:41.114175 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.114137 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" event={"ID":"2a31be42-1cbb-4599-bc04-4529d37f597a","Type":"ContainerStarted","Data":"1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc"} Nov 02 19:59:41.114371 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.114204 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" event={"ID":"2a31be42-1cbb-4599-bc04-4529d37f597a","Type":"ContainerStarted","Data":"0972170cd62697c4d3a6454bd252baef5389b67c8cde708516387202f198a0d7"} Nov 02 19:59:41.114371 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.114235 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:41.116005 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.115982 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" event={"ID":"a32704be-d93d-48c0-b184-1c63d92b8b28","Type":"ContainerStarted","Data":"94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a"} Nov 02 19:59:41.116125 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.116037 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:41.116125 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.116061 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" event={"ID":"a32704be-d93d-48c0-b184-1c63d92b8b28","Type":"ContainerStarted","Data":"dd0ac07574150c1ea8bc34bdc319a5b120bf5b674eb7753a728b9736ba78c1ef"} Nov 02 19:59:41.255724 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.255668 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" podStartSLOduration=1.255653629 podStartE2EDuration="1.255653629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:41.252578122 +0000 UTC m=+679.769350708" watchObservedRunningTime="2026-11-02 19:59:41.255653629 +0000 UTC m=+679.772426213" Nov 02 19:59:41.360882 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:41.360769 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" podStartSLOduration=1.360754238 podStartE2EDuration="1.360754238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:59:41.357684802 +0000 UTC m=+679.874457391" watchObservedRunningTime="2026-11-02 19:59:41.360754238 +0000 UTC m=+679.877526824" Nov 02 19:59:43.156653 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:43.156615 5800 generic.go:356] "Generic (PLEG): container finished" podID="38c5749c-2a39-446d-a037-71501cd76e10" containerID="21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1" exitCode=0 Nov 02 19:59:43.157237 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:43.156693 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" event={"ID":"38c5749c-2a39-446d-a037-71501cd76e10","Type":"ContainerDied","Data":"21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1"} Nov 02 19:59:43.157237 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:43.157160 5800 scope.go:122] "RemoveContainer" containerID="21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1" Nov 02 19:59:43.158581 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:43.158547 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:43.158675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:43.158626 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:43.158675 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:43.158664 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:44.078055 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:44.078023 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 19:59:44.078844 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:44.078821 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 19:59:44.179837 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:44.179798 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" event={"ID":"38c5749c-2a39-446d-a037-71501cd76e10","Type":"ContainerStarted","Data":"916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53"} Nov 02 19:59:44.677559 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:44.677407 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:44.677879 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:44.677848 5800 patch_prober.go:29] interesting pod/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:59:44.677995 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:44.677900 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 19:59:45.676800 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:45.676755 5800 patch_prober.go:29] interesting pod/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:59:45.677323 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:45.676825 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 19:59:46.677729 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:46.677688 5800 patch_prober.go:29] interesting pod/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:59:46.678360 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:46.678325 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 19:59:46.952364 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:46.951394 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:47.676718 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:47.676678 5800 patch_prober.go:29] interesting pod/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:59:47.677030 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:47.676995 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 19:59:48.676063 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:48.676030 5800 patch_prober.go:29] interesting pod/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 02 19:59:48.676476 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:48.676092 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 02 19:59:49.679973 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:49.679938 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:49.681032 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:49.681012 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 19:59:50.001483 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:50.001435 5800 patch_prober.go:29] interesting pod/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:59:50.001657 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:50.001496 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:59:51.243716 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:51.242808 5800 generic.go:356] "Generic (PLEG): container finished" podID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerID="249bed269a32c21e290d3d63f66c7a0a3f2f6b1beb33788405c5bec0ddb8ab29" exitCode=0 Nov 02 19:59:51.243716 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:51.242856 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" event={"ID":"1ba07593-5b6f-4158-9a9a-8c299080b548","Type":"ContainerDied","Data":"249bed269a32c21e290d3d63f66c7a0a3f2f6b1beb33788405c5bec0ddb8ab29"} Nov 02 19:59:51.243716 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:51.243314 5800 scope.go:122] "RemoveContainer" containerID="249bed269a32c21e290d3d63f66c7a0a3f2f6b1beb33788405c5bec0ddb8ab29" Nov 02 19:59:51.244642 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:51.244614 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:51.244693 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:51.244682 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:51.244727 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:51.244711 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 19:59:52.138807 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:52.137787 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 19:59:52.143443 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:52.143416 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 19:59:52.259415 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:52.259385 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" event={"ID":"1ba07593-5b6f-4158-9a9a-8c299080b548","Type":"ContainerStarted","Data":"d045033c70964e9a20fbfa97ab7a016d25fadea46c7f0e7546ea469a9b4d30c3"} Nov 02 19:59:52.576962 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:52.576926 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:52.577254 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:52.577136 5800 patch_prober.go:29] interesting pod/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:59:52.577384 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:52.577310 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 19:59:52.695580 ip-10-0-134-13 kubenswrapper[5800]: E1102 19:59:52.695543 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 19:59:53.576633 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:53.576598 5800 patch_prober.go:29] interesting pod/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:59:53.577030 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:53.576661 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 19:59:54.577377 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:54.577343 5800 patch_prober.go:29] interesting pod/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:59:54.577855 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:54.577400 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 19:59:55.576864 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:55.576826 5800 patch_prober.go:29] interesting pod/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:59:55.577049 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:55.576888 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 19:59:56.577252 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:56.577214 5800 patch_prober.go:29] interesting pod/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 02 19:59:56.577683 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:56.577309 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 02 19:59:57.580787 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:57.580749 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:57.581232 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:57.581036 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 19:59:57.581665 ip-10-0-134-13 kubenswrapper[5800]: I1102 19:59:57.581648 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 20:00:00.164222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.164180 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42"] Nov 02 20:00:00.169902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.169878 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd"] Nov 02 20:00:00.170044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.169994 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.175349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.175319 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.184532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.184491 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf"] Nov 02 20:00:00.194177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.194148 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.219004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.218973 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:00:00.224231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.224211 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:00:00.224374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.224251 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:00:00.260082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.260034 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c304e374-0992-4fd9-8e22-890df767e0c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.260082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.260095 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.260354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.260156 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d687e2c6-cf45-4a68-b9a4-454acfccb297-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-9pqfd\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.260354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.260216 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6652\" (UniqueName: \"kubernetes.io/projected/d687e2c6-cf45-4a68-b9a4-454acfccb297-kube-api-access-b6652\") pod \"maas-api-key-cleanup-29894160-9pqfd\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.260354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.260261 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fgd\" (UniqueName: \"kubernetes.io/projected/c304e374-0992-4fd9-8e22-890df767e0c7-kube-api-access-s7fgd\") pod \"maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.260521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.260390 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvsb2\" (UniqueName: \"kubernetes.io/projected/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-kube-api-access-zvsb2\") pod \"maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.267999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.267955 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42"] Nov 02 20:00:00.270269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.270237 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd"] Nov 02 20:00:00.304993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.304962 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf"] Nov 02 20:00:00.361473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.361437 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c304e374-0992-4fd9-8e22-890df767e0c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.361473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.361475 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.361703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.361506 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d687e2c6-cf45-4a68-b9a4-454acfccb297-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-9pqfd\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.361703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.361541 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6652\" (UniqueName: \"kubernetes.io/projected/d687e2c6-cf45-4a68-b9a4-454acfccb297-kube-api-access-b6652\") pod \"maas-api-key-cleanup-29894160-9pqfd\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.361703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.361573 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fgd\" (UniqueName: \"kubernetes.io/projected/c304e374-0992-4fd9-8e22-890df767e0c7-kube-api-access-s7fgd\") pod \"maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.361820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.361757 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvsb2\" (UniqueName: \"kubernetes.io/projected/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-kube-api-access-zvsb2\") pod \"maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.362328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.362308 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c304e374-0992-4fd9-8e22-890df767e0c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.362425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.362367 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d687e2c6-cf45-4a68-b9a4-454acfccb297-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894160-9pqfd\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.362425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.362366 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.415516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.415449 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fgd\" (UniqueName: \"kubernetes.io/projected/c304e374-0992-4fd9-8e22-890df767e0c7-kube-api-access-s7fgd\") pod \"maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.415516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.415450 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvsb2\" (UniqueName: \"kubernetes.io/projected/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-kube-api-access-zvsb2\") pod \"maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.415516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.415459 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6652\" (UniqueName: \"kubernetes.io/projected/d687e2c6-cf45-4a68-b9a4-454acfccb297-kube-api-access-b6652\") pod \"maas-api-key-cleanup-29894160-9pqfd\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.494511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.494478 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:00.500755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.500723 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:00.509898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.509871 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:00.873512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.873486 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42"] Nov 02 20:00:00.875909 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:00.875880 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1a97bb_4c37_4f9b_ac1b_1557ee4ae15a.slice/crio-dba66de3959eb8e76a20c5020021aed869c37737da1d04a5dd8427dff9b86146 WatchSource:0}: Error finding container dba66de3959eb8e76a20c5020021aed869c37737da1d04a5dd8427dff9b86146: Status 404 returned error can't find the container with id dba66de3959eb8e76a20c5020021aed869c37737da1d04a5dd8427dff9b86146 Nov 02 20:00:00.920508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.920479 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf"] Nov 02 20:00:00.921244 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:00.921211 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc304e374_0992_4fd9_8e22_890df767e0c7.slice/crio-13946a3147c616dc0b1742ba00071a4106c6540f96d80340372954d0a25e21a0 WatchSource:0}: Error finding container 13946a3147c616dc0b1742ba00071a4106c6540f96d80340372954d0a25e21a0: Status 404 returned error can't find the container with id 13946a3147c616dc0b1742ba00071a4106c6540f96d80340372954d0a25e21a0 Nov 02 20:00:00.922036 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:00.922013 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd687e2c6_cf45_4a68_b9a4_454acfccb297.slice/crio-d4b9664a21d118b81da3a36aec688c10948950f264bbf767c2f7f3793e4d0d78 WatchSource:0}: Error finding container d4b9664a21d118b81da3a36aec688c10948950f264bbf767c2f7f3793e4d0d78: Status 404 returned error can't find the container with id d4b9664a21d118b81da3a36aec688c10948950f264bbf767c2f7f3793e4d0d78 Nov 02 20:00:00.923130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:00.923105 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd"] Nov 02 20:00:01.319478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:01.319438 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" event={"ID":"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a","Type":"ContainerStarted","Data":"dba66de3959eb8e76a20c5020021aed869c37737da1d04a5dd8427dff9b86146"} Nov 02 20:00:01.320709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:01.320686 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerStarted","Data":"13946a3147c616dc0b1742ba00071a4106c6540f96d80340372954d0a25e21a0"} Nov 02 20:00:01.321924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:01.321902 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" event={"ID":"d687e2c6-cf45-4a68-b9a4-454acfccb297","Type":"ContainerStarted","Data":"d4b9664a21d118b81da3a36aec688c10948950f264bbf767c2f7f3793e4d0d78"} Nov 02 20:00:06.357614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:06.357516 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" event={"ID":"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a","Type":"ContainerStarted","Data":"2740dc1e5a06e0d410d144d9690d03ab04ac51b3e321515af1d9eda4519b03f0"} Nov 02 20:00:06.359243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:06.359216 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerStarted","Data":"88123fa48dbc1cec664618cb980b43683adc95432f7a974ec3427a6d5d09d15f"} Nov 02 20:00:06.463698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:06.463641 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" podStartSLOduration=1.61906572 podStartE2EDuration="6.463628333s" totalImagesPullingTime="4.844562613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:06.461912362 +0000 UTC m=+704.978684947" watchObservedRunningTime="2026-11-02 20:00:06.463628333 +0000 UTC m=+704.980400919" Nov 02 20:00:06.567453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:06.567403 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" podStartSLOduration=1.733324819 podStartE2EDuration="6.56738947s" totalImagesPullingTime="4.834064651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:06.567270249 +0000 UTC m=+705.084042835" watchObservedRunningTime="2026-11-02 20:00:06.56738947 +0000 UTC m=+705.084162054" Nov 02 20:00:07.367240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.367155 5800 generic.go:356] "Generic (PLEG): container finished" podID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerID="2740dc1e5a06e0d410d144d9690d03ab04ac51b3e321515af1d9eda4519b03f0" exitCode=60 Nov 02 20:00:07.367698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.367243 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" event={"ID":"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a","Type":"ContainerDied","Data":"2740dc1e5a06e0d410d144d9690d03ab04ac51b3e321515af1d9eda4519b03f0"} Nov 02 20:00:07.367698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.367487 5800 scope.go:122] "RemoveContainer" containerID="2740dc1e5a06e0d410d144d9690d03ab04ac51b3e321515af1d9eda4519b03f0" Nov 02 20:00:07.368851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.368832 5800 generic.go:356] "Generic (PLEG): container finished" podID="c304e374-0992-4fd9-8e22-890df767e0c7" containerID="88123fa48dbc1cec664618cb980b43683adc95432f7a974ec3427a6d5d09d15f" exitCode=60 Nov 02 20:00:07.368959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.368918 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerDied","Data":"88123fa48dbc1cec664618cb980b43683adc95432f7a974ec3427a6d5d09d15f"} Nov 02 20:00:07.369126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.369110 5800 scope.go:122] "RemoveContainer" containerID="88123fa48dbc1cec664618cb980b43683adc95432f7a974ec3427a6d5d09d15f" Nov 02 20:00:07.370654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.370629 5800 generic.go:356] "Generic (PLEG): container finished" podID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerID="8dfdd7ad13cee64ce307f990d7c7c8a6721689e77e4541d2f21e6c60874262da" exitCode=60 Nov 02 20:00:07.371132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.370727 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" event={"ID":"d687e2c6-cf45-4a68-b9a4-454acfccb297","Type":"ContainerDied","Data":"8dfdd7ad13cee64ce307f990d7c7c8a6721689e77e4541d2f21e6c60874262da"} Nov 02 20:00:07.378425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:07.371518 5800 scope.go:122] "RemoveContainer" containerID="8dfdd7ad13cee64ce307f990d7c7c8a6721689e77e4541d2f21e6c60874262da" Nov 02 20:00:08.380357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:08.380320 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" event={"ID":"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a","Type":"ContainerStarted","Data":"e6d83cb34b8a2f9ba669fcaead0841a7f7cc8c065eb5394a57156a7af25984c0"} Nov 02 20:00:08.382226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:08.382199 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerStarted","Data":"59ad7a8a2bb1c364f0f380008f8e67155b2cc179d48190f0ebf733bd3d8e99d2"} Nov 02 20:00:08.383916 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:08.383894 5800 generic.go:356] "Generic (PLEG): container finished" podID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerID="72398b9eda0d0b8d3770ff2a030824a6c27682f1b330ca92d6c487d564eda55d" exitCode=60 Nov 02 20:00:08.384033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:08.383942 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" event={"ID":"d687e2c6-cf45-4a68-b9a4-454acfccb297","Type":"ContainerDied","Data":"72398b9eda0d0b8d3770ff2a030824a6c27682f1b330ca92d6c487d564eda55d"} Nov 02 20:00:08.384033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:08.383980 5800 scope.go:122] "RemoveContainer" containerID="8dfdd7ad13cee64ce307f990d7c7c8a6721689e77e4541d2f21e6c60874262da" Nov 02 20:00:08.384143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:08.384125 5800 scope.go:122] "RemoveContainer" containerID="72398b9eda0d0b8d3770ff2a030824a6c27682f1b330ca92d6c487d564eda55d" Nov 02 20:00:08.384387 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:08.384368 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-9pqfd_odh-ai-gateway-infra(d687e2c6-cf45-4a68-b9a4-454acfccb297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" Nov 02 20:00:09.392905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.392872 5800 generic.go:356] "Generic (PLEG): container finished" podID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerID="e6d83cb34b8a2f9ba669fcaead0841a7f7cc8c065eb5394a57156a7af25984c0" exitCode=60 Nov 02 20:00:09.393413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.392956 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" event={"ID":"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a","Type":"ContainerDied","Data":"e6d83cb34b8a2f9ba669fcaead0841a7f7cc8c065eb5394a57156a7af25984c0"} Nov 02 20:00:09.393413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.393033 5800 scope.go:122] "RemoveContainer" containerID="2740dc1e5a06e0d410d144d9690d03ab04ac51b3e321515af1d9eda4519b03f0" Nov 02 20:00:09.393413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.393196 5800 scope.go:122] "RemoveContainer" containerID="e6d83cb34b8a2f9ba669fcaead0841a7f7cc8c065eb5394a57156a7af25984c0" Nov 02 20:00:09.393531 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:09.393500 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42_odh-ai-gateway-infra(af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" Nov 02 20:00:09.394961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.394939 5800 generic.go:356] "Generic (PLEG): container finished" podID="c304e374-0992-4fd9-8e22-890df767e0c7" containerID="59ad7a8a2bb1c364f0f380008f8e67155b2cc179d48190f0ebf733bd3d8e99d2" exitCode=60 Nov 02 20:00:09.395069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.395014 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerDied","Data":"59ad7a8a2bb1c364f0f380008f8e67155b2cc179d48190f0ebf733bd3d8e99d2"} Nov 02 20:00:09.395225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.395208 5800 scope.go:122] "RemoveContainer" containerID="59ad7a8a2bb1c364f0f380008f8e67155b2cc179d48190f0ebf733bd3d8e99d2" Nov 02 20:00:09.395472 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:09.395449 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf_odh-ai-gateway-infra(c304e374-0992-4fd9-8e22-890df767e0c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" Nov 02 20:00:09.396794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.396775 5800 scope.go:122] "RemoveContainer" containerID="72398b9eda0d0b8d3770ff2a030824a6c27682f1b330ca92d6c487d564eda55d" Nov 02 20:00:09.397010 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:09.396993 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-9pqfd_odh-ai-gateway-infra(d687e2c6-cf45-4a68-b9a4-454acfccb297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" Nov 02 20:00:09.405252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:09.405230 5800 scope.go:122] "RemoveContainer" containerID="88123fa48dbc1cec664618cb980b43683adc95432f7a974ec3427a6d5d09d15f" Nov 02 20:00:10.404483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:10.404453 5800 scope.go:122] "RemoveContainer" containerID="e6d83cb34b8a2f9ba669fcaead0841a7f7cc8c065eb5394a57156a7af25984c0" Nov 02 20:00:10.404915 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:10.404739 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42_odh-ai-gateway-infra(af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" Nov 02 20:00:10.405977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:10.405953 5800 scope.go:122] "RemoveContainer" containerID="59ad7a8a2bb1c364f0f380008f8e67155b2cc179d48190f0ebf733bd3d8e99d2" Nov 02 20:00:10.406179 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:10.406161 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf_odh-ai-gateway-infra(c304e374-0992-4fd9-8e22-890df767e0c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" Nov 02 20:00:14.572411 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.572374 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4"] Nov 02 20:00:14.584942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.584909 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.625247 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.625219 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.625433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.625307 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmrk\" (UniqueName: \"kubernetes.io/projected/8437c557-a9a7-47cb-a35f-5857e21e923b-kube-api-access-5gmrk\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.625433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.625395 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.633673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.633649 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:00:14.682809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.682779 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4"] Nov 02 20:00:14.726785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.726752 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.726965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.726795 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmrk\" (UniqueName: \"kubernetes.io/projected/8437c557-a9a7-47cb-a35f-5857e21e923b-kube-api-access-5gmrk\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.726965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.726814 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.727488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.727464 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.729152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.729124 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.780215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.780181 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmrk\" (UniqueName: \"kubernetes.io/projected/8437c557-a9a7-47cb-a35f-5857e21e923b-kube-api-access-5gmrk\") pod \"maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:14.897681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:14.897593 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:15.264927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:15.264900 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4"] Nov 02 20:00:15.266461 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:15.266427 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8437c557_a9a7_47cb_a35f_5857e21e923b.slice/crio-4da99db693579615b2e9608f5ec4f1e41f05b14d93877eaae50a3dad2d9ff901 WatchSource:0}: Error finding container 4da99db693579615b2e9608f5ec4f1e41f05b14d93877eaae50a3dad2d9ff901: Status 404 returned error can't find the container with id 4da99db693579615b2e9608f5ec4f1e41f05b14d93877eaae50a3dad2d9ff901 Nov 02 20:00:15.435218 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:15.435185 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" event={"ID":"8437c557-a9a7-47cb-a35f-5857e21e923b","Type":"ContainerStarted","Data":"4da99db693579615b2e9608f5ec4f1e41f05b14d93877eaae50a3dad2d9ff901"} Nov 02 20:00:15.932417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:15.932348 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2"] Nov 02 20:00:16.448412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:16.448371 5800 generic.go:356] "Generic (PLEG): container finished" podID="8437c557-a9a7-47cb-a35f-5857e21e923b" containerID="0653ee5b2ce397c8de46b108a8e8903ba04396e230fd11fa11fda26e54cc11ee" exitCode=0 Nov 02 20:00:16.449038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:16.448533 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" event={"ID":"8437c557-a9a7-47cb-a35f-5857e21e923b","Type":"ContainerDied","Data":"0653ee5b2ce397c8de46b108a8e8903ba04396e230fd11fa11fda26e54cc11ee"} Nov 02 20:00:17.612404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.612378 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:17.658569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.658537 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-xks-ca\") pod \"8437c557-a9a7-47cb-a35f-5857e21e923b\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " Nov 02 20:00:17.658752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.658578 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-openshift-ca\") pod \"8437c557-a9a7-47cb-a35f-5857e21e923b\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " Nov 02 20:00:17.658752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.658620 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8437c557-a9a7-47cb-a35f-5857e21e923b-kube-api-access-5gmrk\" (UniqueName: \"kubernetes.io/projected/8437c557-a9a7-47cb-a35f-5857e21e923b-kube-api-access-5gmrk\") pod \"8437c557-a9a7-47cb-a35f-5857e21e923b\" (UID: \"8437c557-a9a7-47cb-a35f-5857e21e923b\") " Nov 02 20:00:17.658959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.658936 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-openshift-ca" pod "8437c557-a9a7-47cb-a35f-5857e21e923b" (UID: "8437c557-a9a7-47cb-a35f-5857e21e923b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:17.659063 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.659040 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:17.660791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.660766 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-xks-ca" pod "8437c557-a9a7-47cb-a35f-5857e21e923b" (UID: "8437c557-a9a7-47cb-a35f-5857e21e923b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:00:17.660894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.660835 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8437c557-a9a7-47cb-a35f-5857e21e923b-kube-api-access-5gmrk" pod "8437c557-a9a7-47cb-a35f-5857e21e923b" (UID: "8437c557-a9a7-47cb-a35f-5857e21e923b"). InnerVolumeSpecName "kube-api-access-5gmrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:17.760406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.760376 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8437c557-a9a7-47cb-a35f-5857e21e923b-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:17.760406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:17.760405 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gmrk\" (UniqueName: \"kubernetes.io/projected/8437c557-a9a7-47cb-a35f-5857e21e923b-kube-api-access-5gmrk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:18.461798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:18.461766 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" Nov 02 20:00:18.461980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:18.461797 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4" event={"ID":"8437c557-a9a7-47cb-a35f-5857e21e923b","Type":"ContainerDied","Data":"4da99db693579615b2e9608f5ec4f1e41f05b14d93877eaae50a3dad2d9ff901"} Nov 02 20:00:18.461980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:18.461848 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da99db693579615b2e9608f5ec4f1e41f05b14d93877eaae50a3dad2d9ff901" Nov 02 20:00:20.933513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:20.933472 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" podUID="8a398715-373f-4f62-83aa-e1629a53806f" containerName="payload-processing" containerID="cri-o://2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5" gracePeriod=25 Nov 02 20:00:21.038480 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:21.038453 5800 logging.go:55] [core] [Channel #188 SubChannel #189]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.61:50052", ServerName: "10.132.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.61:50052: connect: connection refused" Nov 02 20:00:21.185665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.185605 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 20:00:21.295855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.295827 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a398715-373f-4f62-83aa-e1629a53806f-kube-api-access-jdmj5\" (UniqueName: \"kubernetes.io/projected/8a398715-373f-4f62-83aa-e1629a53806f-kube-api-access-jdmj5\") pod \"8a398715-373f-4f62-83aa-e1629a53806f\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " Nov 02 20:00:21.296038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.295908 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a398715-373f-4f62-83aa-e1629a53806f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a398715-373f-4f62-83aa-e1629a53806f-plugins-config-volume\") pod \"8a398715-373f-4f62-83aa-e1629a53806f\" (UID: \"8a398715-373f-4f62-83aa-e1629a53806f\") " Nov 02 20:00:21.296379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.296354 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a398715-373f-4f62-83aa-e1629a53806f-plugins-config-volume" pod "8a398715-373f-4f62-83aa-e1629a53806f" (UID: "8a398715-373f-4f62-83aa-e1629a53806f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:21.298176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.298141 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a398715-373f-4f62-83aa-e1629a53806f-kube-api-access-jdmj5" pod "8a398715-373f-4f62-83aa-e1629a53806f" (UID: "8a398715-373f-4f62-83aa-e1629a53806f"). InnerVolumeSpecName "kube-api-access-jdmj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:21.397821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.397783 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmj5\" (UniqueName: \"kubernetes.io/projected/8a398715-373f-4f62-83aa-e1629a53806f-kube-api-access-jdmj5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:21.397821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.397825 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a398715-373f-4f62-83aa-e1629a53806f-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:21.480115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.480034 5800 generic.go:356] "Generic (PLEG): container finished" podID="8a398715-373f-4f62-83aa-e1629a53806f" containerID="2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5" exitCode=0 Nov 02 20:00:21.480115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.480097 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" Nov 02 20:00:21.480306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.480118 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" event={"ID":"8a398715-373f-4f62-83aa-e1629a53806f","Type":"ContainerDied","Data":"2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5"} Nov 02 20:00:21.480306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.480170 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" event={"ID":"8a398715-373f-4f62-83aa-e1629a53806f","Type":"ContainerDied","Data":"8de10cb0ba58fcacac311efb69fec82326bbaf605bfc380a9e6b9fe09a292c79"} Nov 02 20:00:21.480306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.480192 5800 scope.go:122] "RemoveContainer" containerID="2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5" Nov 02 20:00:21.489668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.489652 5800 scope.go:122] "RemoveContainer" containerID="2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5" Nov 02 20:00:21.489953 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:21.489932 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5\": container with ID starting with 2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5 not found: ID does not exist" containerID="2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5" Nov 02 20:00:21.490006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.489962 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5"} err="failed to get container status \"2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5\": rpc error: code = NotFound desc = could not find container \"2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5\": container with ID starting with 2afa3e5737bee403dbf9a293e6ae4a55257281ef4441a32428669918801f39a5 not found: ID does not exist" Nov 02 20:00:21.614371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.614336 5800 scope.go:122] "RemoveContainer" containerID="59ad7a8a2bb1c364f0f380008f8e67155b2cc179d48190f0ebf733bd3d8e99d2" Nov 02 20:00:21.614566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.614511 5800 scope.go:122] "RemoveContainer" containerID="72398b9eda0d0b8d3770ff2a030824a6c27682f1b330ca92d6c487d564eda55d" Nov 02 20:00:21.614662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.614644 5800 scope.go:122] "RemoveContainer" containerID="e6d83cb34b8a2f9ba669fcaead0841a7f7cc8c065eb5394a57156a7af25984c0" Nov 02 20:00:21.636635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.636494 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2"] Nov 02 20:00:21.642122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:21.642094 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2"] Nov 02 20:00:22.038751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.038703 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-4db200eb-74484ffcb5-gvtr2" podUID="8a398715-373f-4f62-83aa-e1629a53806f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.61:50052\" within 1s: context deadline exceeded" Nov 02 20:00:22.488938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.488907 5800 generic.go:356] "Generic (PLEG): container finished" podID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerID="fb89ece7f822a0f4be6c90226e2e49ce09976306b9cb4445e3345dbb2461a4ed" exitCode=60 Nov 02 20:00:22.489111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.488977 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" event={"ID":"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a","Type":"ContainerDied","Data":"fb89ece7f822a0f4be6c90226e2e49ce09976306b9cb4445e3345dbb2461a4ed"} Nov 02 20:00:22.489111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.489064 5800 scope.go:122] "RemoveContainer" containerID="e6d83cb34b8a2f9ba669fcaead0841a7f7cc8c065eb5394a57156a7af25984c0" Nov 02 20:00:22.489509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.489476 5800 scope.go:122] "RemoveContainer" containerID="fb89ece7f822a0f4be6c90226e2e49ce09976306b9cb4445e3345dbb2461a4ed" Nov 02 20:00:22.489792 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:22.489772 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42_odh-ai-gateway-infra(af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" Nov 02 20:00:22.490990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.490968 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerStarted","Data":"105b419e40cbc1ca9ceb14325354bdeab89ba46aedf1855172df8fbf9b504ebe"} Nov 02 20:00:22.492628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.492609 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" event={"ID":"d687e2c6-cf45-4a68-b9a4-454acfccb297","Type":"ContainerStarted","Data":"a8b7c41c89615e8b7972cd275a1358e3afde34981109b7fb476f0df337df1863"} Nov 02 20:00:22.703763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:22.703658 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" podStartSLOduration=17.507264754 podStartE2EDuration="22.70364355s" totalImagesPullingTime="5.196378796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:22.700389713 +0000 UTC m=+721.217162295" watchObservedRunningTime="2026-11-02 20:00:22.70364355 +0000 UTC m=+721.220416175" Nov 02 20:00:22.727258 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:22.727217 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:00:23.100261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.100226 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf"] Nov 02 20:00:23.199768 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.199734 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw"] Nov 02 20:00:23.520678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.520637 5800 generic.go:356] "Generic (PLEG): container finished" podID="c304e374-0992-4fd9-8e22-890df767e0c7" containerID="105b419e40cbc1ca9ceb14325354bdeab89ba46aedf1855172df8fbf9b504ebe" exitCode=60 Nov 02 20:00:23.520956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.520928 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerDied","Data":"105b419e40cbc1ca9ceb14325354bdeab89ba46aedf1855172df8fbf9b504ebe"} Nov 02 20:00:23.521040 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.521002 5800 scope.go:122] "RemoveContainer" containerID="59ad7a8a2bb1c364f0f380008f8e67155b2cc179d48190f0ebf733bd3d8e99d2" Nov 02 20:00:23.521373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.521356 5800 scope.go:122] "RemoveContainer" containerID="105b419e40cbc1ca9ceb14325354bdeab89ba46aedf1855172df8fbf9b504ebe" Nov 02 20:00:23.521616 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:23.521594 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf_odh-ai-gateway-infra(c304e374-0992-4fd9-8e22-890df767e0c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" Nov 02 20:00:23.526698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.526665 5800 generic.go:356] "Generic (PLEG): container finished" podID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerID="a8b7c41c89615e8b7972cd275a1358e3afde34981109b7fb476f0df337df1863" exitCode=60 Nov 02 20:00:23.526781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.526739 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" event={"ID":"d687e2c6-cf45-4a68-b9a4-454acfccb297","Type":"ContainerDied","Data":"a8b7c41c89615e8b7972cd275a1358e3afde34981109b7fb476f0df337df1863"} Nov 02 20:00:23.527185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.527167 5800 scope.go:122] "RemoveContainer" containerID="a8b7c41c89615e8b7972cd275a1358e3afde34981109b7fb476f0df337df1863" Nov 02 20:00:23.527549 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:23.527524 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894160-9pqfd_odh-ai-gateway-infra(d687e2c6-cf45-4a68-b9a4-454acfccb297)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" Nov 02 20:00:23.536944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.536929 5800 scope.go:122] "RemoveContainer" containerID="72398b9eda0d0b8d3770ff2a030824a6c27682f1b330ca92d6c487d564eda55d" Nov 02 20:00:23.606571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.606535 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42"] Nov 02 20:00:23.629153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.628754 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a398715-373f-4f62-83aa-e1629a53806f" path="/var/lib/kubelet/pods/8a398715-373f-4f62-83aa-e1629a53806f/volumes" Nov 02 20:00:23.710767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.710731 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd"] Nov 02 20:00:23.767037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.767012 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:23.815142 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.815066 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf"] Nov 02 20:00:23.826723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.826695 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-kube-api-access-zvsb2\" (UniqueName: \"kubernetes.io/projected/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-kube-api-access-zvsb2\") pod \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " Nov 02 20:00:23.826899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.826745 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-maas-api-service-ca\") pod \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\" (UID: \"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a\") " Nov 02 20:00:23.827114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.827090 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-maas-api-service-ca" pod "af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" (UID: "af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:23.828797 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.828777 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-kube-api-access-zvsb2" pod "af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" (UID: "af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a"). InnerVolumeSpecName "kube-api-access-zvsb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:23.928345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.928312 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvsb2\" (UniqueName: \"kubernetes.io/projected/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-kube-api-access-zvsb2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:23.928345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:23.928342 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:24.534569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.534539 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" Nov 02 20:00:24.535053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.534539 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42" event={"ID":"af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a","Type":"ContainerDied","Data":"dba66de3959eb8e76a20c5020021aed869c37737da1d04a5dd8427dff9b86146"} Nov 02 20:00:24.535053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.534675 5800 scope.go:122] "RemoveContainer" containerID="fb89ece7f822a0f4be6c90226e2e49ce09976306b9cb4445e3345dbb2461a4ed" Nov 02 20:00:24.723818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.723789 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:24.726179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.726154 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:24.792808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.792722 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42"] Nov 02 20:00:24.798429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.798403 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0d19e8c-29894160-hcn42"] Nov 02 20:00:24.836777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.836740 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c304e374-0992-4fd9-8e22-890df767e0c7-kube-api-access-s7fgd\" (UniqueName: \"kubernetes.io/projected/c304e374-0992-4fd9-8e22-890df767e0c7-kube-api-access-s7fgd\") pod \"c304e374-0992-4fd9-8e22-890df767e0c7\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " Nov 02 20:00:24.836948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.836812 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c304e374-0992-4fd9-8e22-890df767e0c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c304e374-0992-4fd9-8e22-890df767e0c7-maas-api-service-ca\") pod \"c304e374-0992-4fd9-8e22-890df767e0c7\" (UID: \"c304e374-0992-4fd9-8e22-890df767e0c7\") " Nov 02 20:00:24.836948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.836922 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d687e2c6-cf45-4a68-b9a4-454acfccb297-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d687e2c6-cf45-4a68-b9a4-454acfccb297-maas-api-service-ca\") pod \"d687e2c6-cf45-4a68-b9a4-454acfccb297\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " Nov 02 20:00:24.837034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.836997 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d687e2c6-cf45-4a68-b9a4-454acfccb297-kube-api-access-b6652\" (UniqueName: \"kubernetes.io/projected/d687e2c6-cf45-4a68-b9a4-454acfccb297-kube-api-access-b6652\") pod \"d687e2c6-cf45-4a68-b9a4-454acfccb297\" (UID: \"d687e2c6-cf45-4a68-b9a4-454acfccb297\") " Nov 02 20:00:24.837211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.837183 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c304e374-0992-4fd9-8e22-890df767e0c7-maas-api-service-ca" pod "c304e374-0992-4fd9-8e22-890df767e0c7" (UID: "c304e374-0992-4fd9-8e22-890df767e0c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:24.837425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.837397 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d687e2c6-cf45-4a68-b9a4-454acfccb297-maas-api-service-ca" pod "d687e2c6-cf45-4a68-b9a4-454acfccb297" (UID: "d687e2c6-cf45-4a68-b9a4-454acfccb297"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:24.837545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.837523 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c304e374-0992-4fd9-8e22-890df767e0c7-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:24.837613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.837550 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d687e2c6-cf45-4a68-b9a4-454acfccb297-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:24.839071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.839043 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d687e2c6-cf45-4a68-b9a4-454acfccb297-kube-api-access-b6652" pod "d687e2c6-cf45-4a68-b9a4-454acfccb297" (UID: "d687e2c6-cf45-4a68-b9a4-454acfccb297"). InnerVolumeSpecName "kube-api-access-b6652". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:24.839159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.839117 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c304e374-0992-4fd9-8e22-890df767e0c7-kube-api-access-s7fgd" pod "c304e374-0992-4fd9-8e22-890df767e0c7" (UID: "c304e374-0992-4fd9-8e22-890df767e0c7"). InnerVolumeSpecName "kube-api-access-s7fgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:24.938573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.938531 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7fgd\" (UniqueName: \"kubernetes.io/projected/c304e374-0992-4fd9-8e22-890df767e0c7-kube-api-access-s7fgd\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:24.938573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:24.938564 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6652\" (UniqueName: \"kubernetes.io/projected/d687e2c6-cf45-4a68-b9a4-454acfccb297-kube-api-access-b6652\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:25.412182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.412146 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb"] Nov 02 20:00:25.513935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.513899 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2"] Nov 02 20:00:25.546101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.546074 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" Nov 02 20:00:25.546554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.546112 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd" event={"ID":"d687e2c6-cf45-4a68-b9a4-454acfccb297","Type":"ContainerDied","Data":"d4b9664a21d118b81da3a36aec688c10948950f264bbf767c2f7f3793e4d0d78"} Nov 02 20:00:25.546554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.546174 5800 scope.go:122] "RemoveContainer" containerID="a8b7c41c89615e8b7972cd275a1358e3afde34981109b7fb476f0df337df1863" Nov 02 20:00:25.548596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.548573 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" event={"ID":"c304e374-0992-4fd9-8e22-890df767e0c7","Type":"ContainerDied","Data":"13946a3147c616dc0b1742ba00071a4106c6540f96d80340372954d0a25e21a0"} Nov 02 20:00:25.548716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.548627 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf" Nov 02 20:00:25.556964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.556945 5800 scope.go:122] "RemoveContainer" containerID="105b419e40cbc1ca9ceb14325354bdeab89ba46aedf1855172df8fbf9b504ebe" Nov 02 20:00:25.621953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.621913 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" path="/var/lib/kubelet/pods/af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a/volumes" Nov 02 20:00:25.733977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.733895 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf"] Nov 02 20:00:25.737708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.737684 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-172adb07-29894160-ndwnf"] Nov 02 20:00:25.839827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.839796 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd"] Nov 02 20:00:25.842613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:25.842588 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894160-9pqfd"] Nov 02 20:00:27.624377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:27.624340 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" path="/var/lib/kubelet/pods/c304e374-0992-4fd9-8e22-890df767e0c7/volumes" Nov 02 20:00:27.625008 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:27.624885 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" path="/var/lib/kubelet/pods/d687e2c6-cf45-4a68-b9a4-454acfccb297/volumes" Nov 02 20:00:28.100982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.100934 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" podUID="4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" containerName="payload-processing" containerID="cri-o://fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d" gracePeriod=25 Nov 02 20:00:28.200528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.200484 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" podUID="0681b630-2424-48af-98df-b04c631a60b8" containerName="payload-pre-processing" containerID="cri-o://017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f" gracePeriod=25 Nov 02 20:00:28.353536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.353469 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 20:00:28.468792 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.468762 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 20:00:28.473340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.473310 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-kube-api-access-szprp\" (UniqueName: \"kubernetes.io/projected/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-kube-api-access-szprp\") pod \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " Nov 02 20:00:28.473511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.473374 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-plugins-config-volume\") pod \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\" (UID: \"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42\") " Nov 02 20:00:28.473734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.473708 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-plugins-config-volume" pod "4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" (UID: "4990dfd9-5aa3-4154-b130-fb9ac1bd6c42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:28.473836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.473823 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.475538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.475513 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-kube-api-access-szprp" pod "4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" (UID: "4990dfd9-5aa3-4154-b130-fb9ac1bd6c42"). InnerVolumeSpecName "kube-api-access-szprp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:28.570239 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.570196 5800 generic.go:356] "Generic (PLEG): container finished" podID="0681b630-2424-48af-98df-b04c631a60b8" containerID="017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f" exitCode=0 Nov 02 20:00:28.570457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.570265 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" Nov 02 20:00:28.570457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.570275 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" event={"ID":"0681b630-2424-48af-98df-b04c631a60b8","Type":"ContainerDied","Data":"017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f"} Nov 02 20:00:28.570457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.570387 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw" event={"ID":"0681b630-2424-48af-98df-b04c631a60b8","Type":"ContainerDied","Data":"7a61212041b1783b3a66ff487520bd97e2de94d66272ae36f51cc61faa87556e"} Nov 02 20:00:28.570457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.570414 5800 scope.go:122] "RemoveContainer" containerID="017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f" Nov 02 20:00:28.571940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.571916 5800 generic.go:356] "Generic (PLEG): container finished" podID="4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" containerID="fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d" exitCode=0 Nov 02 20:00:28.572050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.571975 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" Nov 02 20:00:28.572050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.571981 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" event={"ID":"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42","Type":"ContainerDied","Data":"fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d"} Nov 02 20:00:28.572169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.572084 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf" event={"ID":"4990dfd9-5aa3-4154-b130-fb9ac1bd6c42","Type":"ContainerDied","Data":"e0edafd4d6854b8d241c35127f448915b752ae6d5a9f04c1858824d3455f24bf"} Nov 02 20:00:28.574136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.574091 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0681b630-2424-48af-98df-b04c631a60b8-kube-api-access-lzncl\" (UniqueName: \"kubernetes.io/projected/0681b630-2424-48af-98df-b04c631a60b8-kube-api-access-lzncl\") pod \"0681b630-2424-48af-98df-b04c631a60b8\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " Nov 02 20:00:28.574294 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.574137 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0681b630-2424-48af-98df-b04c631a60b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0681b630-2424-48af-98df-b04c631a60b8-plugins-config-volume\") pod \"0681b630-2424-48af-98df-b04c631a60b8\" (UID: \"0681b630-2424-48af-98df-b04c631a60b8\") " Nov 02 20:00:28.575339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.575313 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szprp\" (UniqueName: \"kubernetes.io/projected/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42-kube-api-access-szprp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.575541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.575513 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0681b630-2424-48af-98df-b04c631a60b8-plugins-config-volume" pod "0681b630-2424-48af-98df-b04c631a60b8" (UID: "0681b630-2424-48af-98df-b04c631a60b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:28.576877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.576849 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0681b630-2424-48af-98df-b04c631a60b8-kube-api-access-lzncl" pod "0681b630-2424-48af-98df-b04c631a60b8" (UID: "0681b630-2424-48af-98df-b04c631a60b8"). InnerVolumeSpecName "kube-api-access-lzncl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:28.601945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.601915 5800 scope.go:122] "RemoveContainer" containerID="017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f" Nov 02 20:00:28.604472 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:28.604394 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f\": container with ID starting with 017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f not found: ID does not exist" containerID="017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f" Nov 02 20:00:28.604655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.604624 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f"} err="failed to get container status \"017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f\": rpc error: code = NotFound desc = could not find container \"017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f\": container with ID starting with 017a0b74400d539b21f3657155095d4f86df78ef3d34cba3fa2049f1c1a89f8f not found: ID does not exist" Nov 02 20:00:28.604792 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.604775 5800 scope.go:122] "RemoveContainer" containerID="fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d" Nov 02 20:00:28.620398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.620373 5800 scope.go:122] "RemoveContainer" containerID="fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d" Nov 02 20:00:28.620768 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:28.620744 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d\": container with ID starting with fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d not found: ID does not exist" containerID="fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d" Nov 02 20:00:28.620879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.620778 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d"} err="failed to get container status \"fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d\": rpc error: code = NotFound desc = could not find container \"fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d\": container with ID starting with fb6475260065b71020fd489d567b10c0b43abaf1df134d52f93c5c065588111d not found: ID does not exist" Nov 02 20:00:28.674849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.674813 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r"] Nov 02 20:00:28.675391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.675023 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" containerID="cri-o://916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53" gracePeriod=30 Nov 02 20:00:28.675845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.675823 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzncl\" (UniqueName: \"kubernetes.io/projected/0681b630-2424-48af-98df-b04c631a60b8-kube-api-access-lzncl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.675845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.675844 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0681b630-2424-48af-98df-b04c631a60b8-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:28.778488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.778461 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf"] Nov 02 20:00:28.780574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.780550 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-70ebea5b-55bc9d5c97-kx4kf"] Nov 02 20:00:28.991204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.991160 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw"] Nov 02 20:00:28.993209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:28.993188 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-70ebea5b-7955447c59-bt5fw"] Nov 02 20:00:29.620719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:29.620687 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0681b630-2424-48af-98df-b04c631a60b8" path="/var/lib/kubelet/pods/0681b630-2424-48af-98df-b04c631a60b8/volumes" Nov 02 20:00:29.621077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:29.621064 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" path="/var/lib/kubelet/pods/4990dfd9-5aa3-4154-b130-fb9ac1bd6c42/volumes" Nov 02 20:00:30.008691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.008656 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc"] Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009103 5800 cpu_manager.go:414] "containerMap: removing container" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009114 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009124 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009129 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009140 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" containerName="payload-processing" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009146 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" containerName="payload-processing" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009155 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009160 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009173 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0681b630-2424-48af-98df-b04c631a60b8" containerName="payload-pre-processing" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009178 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0681b630-2424-48af-98df-b04c631a60b8" containerName="payload-pre-processing" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009192 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8437c557-a9a7-47cb-a35f-5857e21e923b" containerName="revoke-keys" Nov 02 20:00:30.009192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009197 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8437c557-a9a7-47cb-a35f-5857e21e923b" containerName="revoke-keys" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009207 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009212 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009227 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8a398715-373f-4f62-83aa-e1629a53806f" containerName="payload-processing" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009232 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a398715-373f-4f62-83aa-e1629a53806f" containerName="payload-processing" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009241 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009246 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009253 5800 cpu_manager.go:414] "containerMap: removing container" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009258 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009350 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009357 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009363 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009368 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009375 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4990dfd9-5aa3-4154-b130-fb9ac1bd6c42" containerName="payload-processing" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009383 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0681b630-2424-48af-98df-b04c631a60b8" containerName="payload-pre-processing" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009406 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009412 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8437c557-a9a7-47cb-a35f-5857e21e923b" containerName="revoke-keys" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009418 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009424 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009429 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:30.009798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.009435 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a398715-373f-4f62-83aa-e1629a53806f" containerName="payload-processing" Nov 02 20:00:30.014326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.014304 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.073309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.063408 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-426941-openshift-default-dockercfg-mxlvf\"" Nov 02 20:00:30.073309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.070608 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc"] Nov 02 20:00:30.091552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.091514 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.091720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.091561 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.091720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.091656 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.091720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.091685 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnjf\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-kube-api-access-ttnjf\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.091889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.091825 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.091942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.091883 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.091991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.091963 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.092038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.092019 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.092089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.092039 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.192775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.192738 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.192966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.192827 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.192966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.192862 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.192966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.192905 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.192966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.192929 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.193182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.192974 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.193182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.193008 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnjf\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-kube-api-access-ttnjf\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.193182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.193069 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.193182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.193141 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.193422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.193395 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.193734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.193694 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.193915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.193890 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.194028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.194009 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.194181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.194160 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.195365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.195343 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.195692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.195665 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.246168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.246128 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.246359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.246338 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnjf\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-kube-api-access-ttnjf\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.327266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.327181 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:30.412547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.412442 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" podUID="dec33fbe-1eab-4db3-b85b-e1abb7a464dd" containerName="payload-processing" containerID="cri-o://f3369ce41a2b6a01776708139f9c5b69f977e692a82bf46bf635612d793e67ce" gracePeriod=25 Nov 02 20:00:30.423770 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.423739 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92"] Nov 02 20:00:30.424029 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.423999 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" containerID="cri-o://d045033c70964e9a20fbfa97ab7a016d25fadea46c7f0e7546ea469a9b4d30c3" gracePeriod=30 Nov 02 20:00:30.514687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.514649 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" podUID="41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" containerName="payload-pre-processing" containerID="cri-o://fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08" gracePeriod=25 Nov 02 20:00:30.587782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.587693 5800 generic.go:356] "Generic (PLEG): container finished" podID="dec33fbe-1eab-4db3-b85b-e1abb7a464dd" containerID="f3369ce41a2b6a01776708139f9c5b69f977e692a82bf46bf635612d793e67ce" exitCode=0 Nov 02 20:00:30.587782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.587765 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" event={"ID":"dec33fbe-1eab-4db3-b85b-e1abb7a464dd","Type":"ContainerDied","Data":"f3369ce41a2b6a01776708139f9c5b69f977e692a82bf46bf635612d793e67ce"} Nov 02 20:00:30.681682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.681650 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 20:00:30.743416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.743211 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc"] Nov 02 20:00:30.753232 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:30.753201 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8398ad9_e519_4c49_a40e_1efedaf04d5d.slice/crio-d8a1e9d73cce84b77796c360250c8002b143745bfab4e9b93e31c947c1a77adf WatchSource:0}: Error finding container d8a1e9d73cce84b77796c360250c8002b143745bfab4e9b93e31c947c1a77adf: Status 404 returned error can't find the container with id d8a1e9d73cce84b77796c360250c8002b143745bfab4e9b93e31c947c1a77adf Nov 02 20:00:30.756039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.756004 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:30.756131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.756091 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:30.756266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.756139 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:30.780863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.780841 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 20:00:30.804319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.804253 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-plugins-config-volume\") pod \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " Nov 02 20:00:30.804461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.804342 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-kube-api-access-x9mmg\" (UniqueName: \"kubernetes.io/projected/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-kube-api-access-x9mmg\") pod \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\" (UID: \"dec33fbe-1eab-4db3-b85b-e1abb7a464dd\") " Nov 02 20:00:30.804694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.804665 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-plugins-config-volume" pod "dec33fbe-1eab-4db3-b85b-e1abb7a464dd" (UID: "dec33fbe-1eab-4db3-b85b-e1abb7a464dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:30.804839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.804822 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:30.807446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.807414 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-kube-api-access-x9mmg" pod "dec33fbe-1eab-4db3-b85b-e1abb7a464dd" (UID: "dec33fbe-1eab-4db3-b85b-e1abb7a464dd"). InnerVolumeSpecName "kube-api-access-x9mmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:30.905891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.905799 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-kube-api-access-jq8bv\" (UniqueName: \"kubernetes.io/projected/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-kube-api-access-jq8bv\") pod \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " Nov 02 20:00:30.905891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.905880 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-plugins-config-volume\") pod \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\" (UID: \"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a\") " Nov 02 20:00:30.906342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.906318 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mmg\" (UniqueName: \"kubernetes.io/projected/dec33fbe-1eab-4db3-b85b-e1abb7a464dd-kube-api-access-x9mmg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:30.906724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.906695 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-plugins-config-volume" pod "41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" (UID: "41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:30.909537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:30.909515 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-kube-api-access-jq8bv" pod "41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" (UID: "41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a"). InnerVolumeSpecName "kube-api-access-jq8bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:31.007898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.007867 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8bv\" (UniqueName: \"kubernetes.io/projected/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-kube-api-access-jq8bv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:31.007898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.007895 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:31.595044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.595007 5800 generic.go:356] "Generic (PLEG): container finished" podID="41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" containerID="fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08" exitCode=0 Nov 02 20:00:31.595509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.595066 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" Nov 02 20:00:31.595509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.595087 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" event={"ID":"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a","Type":"ContainerDied","Data":"fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08"} Nov 02 20:00:31.595509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.595175 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2" event={"ID":"41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a","Type":"ContainerDied","Data":"5737ffd9f0d3a90578acb0214fdda31e7ed4a3e813b044605816b2443905ada8"} Nov 02 20:00:31.595509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.595203 5800 scope.go:122] "RemoveContainer" containerID="fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08" Nov 02 20:00:31.596643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.596625 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" Nov 02 20:00:31.596761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.596628 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb" event={"ID":"dec33fbe-1eab-4db3-b85b-e1abb7a464dd","Type":"ContainerDied","Data":"de65a343af8f78ba07bc831d06f0f0099c47d6409c3dd25ccb421582f94d268d"} Nov 02 20:00:31.598578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.598553 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" event={"ID":"b8398ad9-e519-4c49-a40e-1efedaf04d5d","Type":"ContainerStarted","Data":"815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810"} Nov 02 20:00:31.598692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.598600 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" event={"ID":"b8398ad9-e519-4c49-a40e-1efedaf04d5d","Type":"ContainerStarted","Data":"d8a1e9d73cce84b77796c360250c8002b143745bfab4e9b93e31c947c1a77adf"} Nov 02 20:00:31.607414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.607395 5800 scope.go:122] "RemoveContainer" containerID="fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08" Nov 02 20:00:31.615977 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:31.615824 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08\": container with ID starting with fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08 not found: ID does not exist" containerID="fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08" Nov 02 20:00:31.615977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.615864 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08"} err="failed to get container status \"fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08\": rpc error: code = NotFound desc = could not find container \"fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08\": container with ID starting with fee728078e3500bea173abdcf965a5f4125df0a6e310424cc85f00ee4a0faf08 not found: ID does not exist" Nov 02 20:00:31.615977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.615888 5800 scope.go:122] "RemoveContainer" containerID="f3369ce41a2b6a01776708139f9c5b69f977e692a82bf46bf635612d793e67ce" Nov 02 20:00:31.749227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.749187 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb"] Nov 02 20:00:31.754776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.754750 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0d19e8c-6cdc976cf9-j89wb"] Nov 02 20:00:31.795573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.795541 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9"] Nov 02 20:00:31.796021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796007 5800 cpu_manager.go:414] "containerMap: removing container" podUID="dec33fbe-1eab-4db3-b85b-e1abb7a464dd" containerName="payload-processing" Nov 02 20:00:31.796069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796024 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec33fbe-1eab-4db3-b85b-e1abb7a464dd" containerName="payload-processing" Nov 02 20:00:31.796069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796034 5800 cpu_manager.go:414] "containerMap: removing container" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:31.796069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796042 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1a97bb-4c37-4f9b-ac1b-1557ee4ae15a" containerName="cleanup" Nov 02 20:00:31.796069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796048 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:31.796069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796053 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d687e2c6-cf45-4a68-b9a4-454acfccb297" containerName="cleanup" Nov 02 20:00:31.796069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796064 5800 cpu_manager.go:414] "containerMap: removing container" podUID="41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" containerName="payload-pre-processing" Nov 02 20:00:31.796069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796068 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" containerName="payload-pre-processing" Nov 02 20:00:31.796341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796092 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:31.796341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796098 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:31.796341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796168 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c304e374-0992-4fd9-8e22-890df767e0c7" containerName="cleanup" Nov 02 20:00:31.796341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796177 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec33fbe-1eab-4db3-b85b-e1abb7a464dd" containerName="payload-processing" Nov 02 20:00:31.796341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.796184 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" containerName="payload-pre-processing" Nov 02 20:00:31.801358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.801324 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.834968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.833314 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9"] Nov 02 20:00:31.852381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.849888 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-14d090-openshift-default-dockercfg-5wl9l\"" Nov 02 20:00:31.862865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.862833 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2"] Nov 02 20:00:31.866483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.866460 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0d19e8c-5b8bbc68cf-j68z2"] Nov 02 20:00:31.919197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919157 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919233 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsph\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-kube-api-access-rwsph\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919377 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919421 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919449 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919570 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919635 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919664 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.919815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.919703 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:31.974760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:31.974703 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podStartSLOduration=2.974684464 podStartE2EDuration="2.974684464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:31.971384682 +0000 UTC m=+730.488157270" watchObservedRunningTime="2026-11-02 20:00:31.974684464 +0000 UTC m=+730.491457046" Nov 02 20:00:32.021267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021226 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021344 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021382 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021450 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021520 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsph\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-kube-api-access-rwsph\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021576 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021609 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021636 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021694 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.021890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021833 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.022011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021957 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.022011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.021992 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.022107 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.022065 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.022155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.022127 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.024138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.024110 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.024252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.024136 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.092329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.084816 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.092329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.084913 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4"] Nov 02 20:00:32.092329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.087453 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsph\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-kube-api-access-rwsph\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.092329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.090561 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0d19e8c-5xhd4"] Nov 02 20:00:32.123461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.123387 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:32.327834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.327800 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:32.333450 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.333419 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:32.538379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.538347 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9"] Nov 02 20:00:32.542389 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:32.542346 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de6c98b_c7c5_452c_ba6c_87179d8a5127.slice/crio-27ca4d2458a1f10430bd67f8fb16514c37931892924b882ef7e4dca74de7dd72 WatchSource:0}: Error finding container 27ca4d2458a1f10430bd67f8fb16514c37931892924b882ef7e4dca74de7dd72: Status 404 returned error can't find the container with id 27ca4d2458a1f10430bd67f8fb16514c37931892924b882ef7e4dca74de7dd72 Nov 02 20:00:32.545533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.545496 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:32.545601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.545583 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:32.545641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.545626 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:32.607320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.607289 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" event={"ID":"3de6c98b-c7c5-452c-ba6c-87179d8a5127","Type":"ContainerStarted","Data":"27ca4d2458a1f10430bd67f8fb16514c37931892924b882ef7e4dca74de7dd72"} Nov 02 20:00:32.609664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.609645 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:32.610508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:32.610486 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:00:33.627121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:33.627091 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a" path="/var/lib/kubelet/pods/41b39ce6-59a3-4e67-9a40-4ea6f6b56e4a/volumes" Nov 02 20:00:33.627634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:33.627601 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8437c557-a9a7-47cb-a35f-5857e21e923b" path="/var/lib/kubelet/pods/8437c557-a9a7-47cb-a35f-5857e21e923b/volumes" Nov 02 20:00:33.627961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:33.627946 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec33fbe-1eab-4db3-b85b-e1abb7a464dd" path="/var/lib/kubelet/pods/dec33fbe-1eab-4db3-b85b-e1abb7a464dd/volumes" Nov 02 20:00:33.628238 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:33.628222 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" event={"ID":"3de6c98b-c7c5-452c-ba6c-87179d8a5127","Type":"ContainerStarted","Data":"e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12"} Nov 02 20:00:33.731722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:33.731665 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podStartSLOduration=2.731649387 podStartE2EDuration="2.731649387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:33.730415873 +0000 UTC m=+732.247188458" watchObservedRunningTime="2026-11-02 20:00:33.731649387 +0000 UTC m=+732.248421971" Nov 02 20:00:33.965446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:33.965422 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 20:00:34.060879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.060843 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-certs\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.060895 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-kube-api-access-nf9z5\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-kube-api-access-nf9z5\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.060926 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-socket\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.060976 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c5749c-2a39-446d-a037-71501cd76e10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c5749c-2a39-446d-a037-71501cd76e10-istiod-ca-cert\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061028 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38c5749c-2a39-446d-a037-71501cd76e10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c5749c-2a39-446d-a037-71501cd76e10-istio-podinfo\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061054 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-envoy\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061073 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-credential-socket\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061093 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-istio-token\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-istio-token\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061161 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-data\") pod \"38c5749c-2a39-446d-a037-71501cd76e10\" (UID: \"38c5749c-2a39-446d-a037-71501cd76e10\") " Nov 02 20:00:34.061320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061161 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-certs" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:34.061320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061233 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-socket" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:34.061534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061511 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-data" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:34.061594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061550 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c5749c-2a39-446d-a037-71501cd76e10-istiod-ca-cert" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:34.061645 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061623 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.061697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061644 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.061697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061660 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.061799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.061735 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-credential-socket" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:34.063496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.063472 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-envoy" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:34.063496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.063485 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38c5749c-2a39-446d-a037-71501cd76e10-istio-podinfo" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:34.063636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.063502 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-kube-api-access-nf9z5" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "kube-api-access-nf9z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:34.063636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.063526 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-istio-token" pod "38c5749c-2a39-446d-a037-71501cd76e10" (UID: "38c5749c-2a39-446d-a037-71501cd76e10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:34.123958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.123926 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:34.128908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.128883 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:34.162871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.162841 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9z5\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-kube-api-access-nf9z5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.162871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.162867 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38c5749c-2a39-446d-a037-71501cd76e10-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.163056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.162879 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38c5749c-2a39-446d-a037-71501cd76e10-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.163056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.162888 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.163056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.162896 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38c5749c-2a39-446d-a037-71501cd76e10-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.163056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.162905 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c5749c-2a39-446d-a037-71501cd76e10-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:34.627906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.627868 5800 generic.go:356] "Generic (PLEG): container finished" podID="38c5749c-2a39-446d-a037-71501cd76e10" containerID="916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53" exitCode=0 Nov 02 20:00:34.628440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.627933 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" Nov 02 20:00:34.628440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.627946 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" event={"ID":"38c5749c-2a39-446d-a037-71501cd76e10","Type":"ContainerDied","Data":"916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53"} Nov 02 20:00:34.628440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.628021 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r" event={"ID":"38c5749c-2a39-446d-a037-71501cd76e10","Type":"ContainerDied","Data":"8606919c4fd7a8011797b21abceef1faaa16e79a62a5a7a686c55b83cad6fe54"} Nov 02 20:00:34.628440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.628055 5800 scope.go:122] "RemoveContainer" containerID="916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53" Nov 02 20:00:34.628440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.628416 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:34.630174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.630148 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:00:34.639690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.639606 5800 scope.go:122] "RemoveContainer" containerID="21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1" Nov 02 20:00:34.650269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.650244 5800 scope.go:122] "RemoveContainer" containerID="916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53" Nov 02 20:00:34.650615 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:34.650592 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53\": container with ID starting with 916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53 not found: ID does not exist" containerID="916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53" Nov 02 20:00:34.650696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.650626 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53"} err="failed to get container status \"916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53\": rpc error: code = NotFound desc = could not find container \"916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53\": container with ID starting with 916b049e4c0dfc22353fa8d1e0c8f402b59460691bda445fd8b6353c8752ea53 not found: ID does not exist" Nov 02 20:00:34.650696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.650649 5800 scope.go:122] "RemoveContainer" containerID="21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1" Nov 02 20:00:34.650957 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:34.650938 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1\": container with ID starting with 21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1 not found: ID does not exist" containerID="21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1" Nov 02 20:00:34.651018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.650966 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1"} err="failed to get container status \"21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1\": rpc error: code = NotFound desc = could not find container \"21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1\": container with ID starting with 21765d3b0b844eb6624efc91c2c0500236819f694f638d394c8624a9ee4559f1 not found: ID does not exist" Nov 02 20:00:34.890967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.890873 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r"] Nov 02 20:00:34.894843 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:34.894814 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4db200eb-openshift-default-7457499b79-gk52r"] Nov 02 20:00:35.624905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.624869 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c5749c-2a39-446d-a037-71501cd76e10" path="/var/lib/kubelet/pods/38c5749c-2a39-446d-a037-71501cd76e10/volumes" Nov 02 20:00:35.637651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.637616 5800 generic.go:356] "Generic (PLEG): container finished" podID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerID="d045033c70964e9a20fbfa97ab7a016d25fadea46c7f0e7546ea469a9b4d30c3" exitCode=0 Nov 02 20:00:35.638129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.637695 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" event={"ID":"1ba07593-5b6f-4158-9a9a-8c299080b548","Type":"ContainerDied","Data":"d045033c70964e9a20fbfa97ab7a016d25fadea46c7f0e7546ea469a9b4d30c3"} Nov 02 20:00:35.638129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.637753 5800 scope.go:122] "RemoveContainer" containerID="249bed269a32c21e290d3d63f66c7a0a3f2f6b1beb33788405c5bec0ddb8ab29" Nov 02 20:00:35.689299 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.689229 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 20:00:35.880995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.880953 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-data\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881020 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-certs\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881054 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-podinfo\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881123 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-credential-socket\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881160 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-envoy\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881199 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-socket\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881319 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-data" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:35.881414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881343 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-token\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-token\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881385 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-kube-api-access-gscb5\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-kube-api-access-gscb5\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881418 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba07593-5b6f-4158-9a9a-8c299080b548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba07593-5b6f-4158-9a9a-8c299080b548-istiod-ca-cert\") pod \"1ba07593-5b6f-4158-9a9a-8c299080b548\" (UID: \"1ba07593-5b6f-4158-9a9a-8c299080b548\") " Nov 02 20:00:35.881613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881488 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-certs" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:35.881774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881750 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-socket" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:35.881858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881817 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.881858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.881838 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.882133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.882106 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-credential-socket" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:35.883322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.882561 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba07593-5b6f-4158-9a9a-8c299080b548-istiod-ca-cert" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:35.883926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.883898 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-podinfo" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:00:35.884070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.884051 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-kube-api-access-gscb5" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "kube-api-access-gscb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:35.884413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.884393 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-token" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:35.884767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.884746 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-envoy" pod "1ba07593-5b6f-4158-9a9a-8c299080b548" (UID: "1ba07593-5b6f-4158-9a9a-8c299080b548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:00:35.982974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.982886 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.982974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.982918 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.982974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.982929 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.982974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.982937 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gscb5\" (UniqueName: \"kubernetes.io/projected/1ba07593-5b6f-4158-9a9a-8c299080b548-kube-api-access-gscb5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.982974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.982948 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba07593-5b6f-4158-9a9a-8c299080b548-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.982974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.982959 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba07593-5b6f-4158-9a9a-8c299080b548-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:35.982974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:35.982967 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba07593-5b6f-4158-9a9a-8c299080b548-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:36.180428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:36.180397 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr"] Nov 02 20:00:36.238608 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:36.238525 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49"] Nov 02 20:00:36.646379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:36.646336 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" event={"ID":"1ba07593-5b6f-4158-9a9a-8c299080b548","Type":"ContainerDied","Data":"d44f331d74d9dd5e5d0f2b431f9070cfbb3a2c661b88f1a29686c50a150e20da"} Nov 02 20:00:36.646844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:36.646408 5800 scope.go:122] "RemoveContainer" containerID="d045033c70964e9a20fbfa97ab7a016d25fadea46c7f0e7546ea469a9b4d30c3" Nov 02 20:00:36.646844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:36.646352 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92" Nov 02 20:00:36.800668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:36.800629 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92"] Nov 02 20:00:36.818730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:36.818701 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-70ebea5b-openshift-default-7586f7bbf9-9fp92"] Nov 02 20:00:37.622461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:37.622429 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" path="/var/lib/kubelet/pods/1ba07593-5b6f-4158-9a9a-8c299080b548/volumes" Nov 02 20:00:40.157709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.157673 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9"] Nov 02 20:00:40.158188 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158166 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" Nov 02 20:00:40.158188 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158181 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" Nov 02 20:00:40.158354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158196 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" Nov 02 20:00:40.158354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158201 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" Nov 02 20:00:40.158354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158213 5800 cpu_manager.go:414] "containerMap: removing container" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" Nov 02 20:00:40.158354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158218 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" Nov 02 20:00:40.158354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158244 5800 cpu_manager.go:414] "containerMap: removing container" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" Nov 02 20:00:40.158354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158248 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" Nov 02 20:00:40.158634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158413 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" Nov 02 20:00:40.158634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158426 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba07593-5b6f-4158-9a9a-8c299080b548" containerName="istio-proxy" Nov 02 20:00:40.158634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158436 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" Nov 02 20:00:40.158634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.158447 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c5749c-2a39-446d-a037-71501cd76e10" containerName="istio-proxy" Nov 02 20:00:40.166887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.166861 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.214551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.214515 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9"] Nov 02 20:00:40.235035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.234999 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr5n\" (UniqueName: \"kubernetes.io/projected/83840101-e53a-436f-ae00-bcdd34063062-kube-api-access-pmr5n\") pod \"payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.235199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.235079 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83840101-e53a-436f-ae00-bcdd34063062-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.258503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.258456 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-426941\"" Nov 02 20:00:40.258641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.258586 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-426941-dockercfg-bt2hh\"" Nov 02 20:00:40.336546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.336507 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83840101-e53a-436f-ae00-bcdd34063062-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.336749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.336662 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr5n\" (UniqueName: \"kubernetes.io/projected/83840101-e53a-436f-ae00-bcdd34063062-kube-api-access-pmr5n\") pod \"payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.337490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.337464 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83840101-e53a-436f-ae00-bcdd34063062-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.391023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.390994 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr5n\" (UniqueName: \"kubernetes.io/projected/83840101-e53a-436f-ae00-bcdd34063062-kube-api-access-pmr5n\") pod \"payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.481449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.481353 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:40.853078 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:40.852987 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9"] Nov 02 20:00:41.181318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.181194 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" podUID="0b8cbaad-4073-4be8-8ff9-5551b6eca79f" containerName="payload-processing" containerID="cri-o://0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c" gracePeriod=25 Nov 02 20:00:41.239833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.239775 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" podUID="82d780c0-9801-4f46-bd90-e0f102a877d7" containerName="payload-pre-processing" containerID="cri-o://ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133" gracePeriod=25 Nov 02 20:00:41.520197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.520174 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 20:00:41.523610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.523589 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 20:00:41.548310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.548270 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d780c0-9801-4f46-bd90-e0f102a877d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d780c0-9801-4f46-bd90-e0f102a877d7-plugins-config-volume\") pod \"82d780c0-9801-4f46-bd90-e0f102a877d7\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " Nov 02 20:00:41.548465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.548371 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d780c0-9801-4f46-bd90-e0f102a877d7-kube-api-access-6zgqq\" (UniqueName: \"kubernetes.io/projected/82d780c0-9801-4f46-bd90-e0f102a877d7-kube-api-access-6zgqq\") pod \"82d780c0-9801-4f46-bd90-e0f102a877d7\" (UID: \"82d780c0-9801-4f46-bd90-e0f102a877d7\") " Nov 02 20:00:41.548465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.548423 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-kube-api-access-6gmtf\" (UniqueName: \"kubernetes.io/projected/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-kube-api-access-6gmtf\") pod \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " Nov 02 20:00:41.548571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.548471 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-plugins-config-volume\") pod \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\" (UID: \"0b8cbaad-4073-4be8-8ff9-5551b6eca79f\") " Nov 02 20:00:41.548632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.548600 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d780c0-9801-4f46-bd90-e0f102a877d7-plugins-config-volume" pod "82d780c0-9801-4f46-bd90-e0f102a877d7" (UID: "82d780c0-9801-4f46-bd90-e0f102a877d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:41.548837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.548819 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d780c0-9801-4f46-bd90-e0f102a877d7-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:41.548939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.548916 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-plugins-config-volume" pod "0b8cbaad-4073-4be8-8ff9-5551b6eca79f" (UID: "0b8cbaad-4073-4be8-8ff9-5551b6eca79f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:41.550465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.550442 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d780c0-9801-4f46-bd90-e0f102a877d7-kube-api-access-6zgqq" pod "82d780c0-9801-4f46-bd90-e0f102a877d7" (UID: "82d780c0-9801-4f46-bd90-e0f102a877d7"). InnerVolumeSpecName "kube-api-access-6zgqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:41.550613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.550599 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-kube-api-access-6gmtf" pod "0b8cbaad-4073-4be8-8ff9-5551b6eca79f" (UID: "0b8cbaad-4073-4be8-8ff9-5551b6eca79f"). InnerVolumeSpecName "kube-api-access-6gmtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:41.649790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.649759 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgqq\" (UniqueName: \"kubernetes.io/projected/82d780c0-9801-4f46-bd90-e0f102a877d7-kube-api-access-6zgqq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:41.649790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.649791 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gmtf\" (UniqueName: \"kubernetes.io/projected/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-kube-api-access-6gmtf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:41.650026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.649805 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8cbaad-4073-4be8-8ff9-5551b6eca79f-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:41.677986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.677944 5800 generic.go:356] "Generic (PLEG): container finished" podID="0b8cbaad-4073-4be8-8ff9-5551b6eca79f" containerID="0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c" exitCode=0 Nov 02 20:00:41.678149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.678003 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" Nov 02 20:00:41.678149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.678035 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" event={"ID":"0b8cbaad-4073-4be8-8ff9-5551b6eca79f","Type":"ContainerDied","Data":"0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c"} Nov 02 20:00:41.678149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.678092 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr" event={"ID":"0b8cbaad-4073-4be8-8ff9-5551b6eca79f","Type":"ContainerDied","Data":"8dcb404e566c8c84607033282e8d3995a8b29582243c7bf93926b2a59247eeae"} Nov 02 20:00:41.678149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.678115 5800 scope.go:122] "RemoveContainer" containerID="0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c" Nov 02 20:00:41.679811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.679789 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" event={"ID":"83840101-e53a-436f-ae00-bcdd34063062","Type":"ContainerStarted","Data":"ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4"} Nov 02 20:00:41.679963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.679833 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" event={"ID":"83840101-e53a-436f-ae00-bcdd34063062","Type":"ContainerStarted","Data":"873043528cd33d5e30ce8cafd6dcf9e7ef9a06de9028f1bc5aed68d9642ab490"} Nov 02 20:00:41.679963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.679918 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:41.681748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.681726 5800 generic.go:356] "Generic (PLEG): container finished" podID="82d780c0-9801-4f46-bd90-e0f102a877d7" containerID="ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133" exitCode=0 Nov 02 20:00:41.681851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.681758 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" event={"ID":"82d780c0-9801-4f46-bd90-e0f102a877d7","Type":"ContainerDied","Data":"ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133"} Nov 02 20:00:41.681851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.681781 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" Nov 02 20:00:41.681851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.681786 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49" event={"ID":"82d780c0-9801-4f46-bd90-e0f102a877d7","Type":"ContainerDied","Data":"3ca39faac4d8c6ccf966e947f335cd88babda5a14ab82e6e2f22dc527f0e8840"} Nov 02 20:00:41.688773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.688752 5800 scope.go:122] "RemoveContainer" containerID="0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c" Nov 02 20:00:41.689060 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:41.689040 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c\": container with ID starting with 0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c not found: ID does not exist" containerID="0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c" Nov 02 20:00:41.689115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.689068 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c"} err="failed to get container status \"0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c\": rpc error: code = NotFound desc = could not find container \"0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c\": container with ID starting with 0ad0646019b9b2bb33c4a445bff5cd872b9564955339848196fbb74ac4a58f0c not found: ID does not exist" Nov 02 20:00:41.689115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.689085 5800 scope.go:122] "RemoveContainer" containerID="ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133" Nov 02 20:00:41.697643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.697620 5800 scope.go:122] "RemoveContainer" containerID="ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133" Nov 02 20:00:41.697914 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:41.697894 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133\": container with ID starting with ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133 not found: ID does not exist" containerID="ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133" Nov 02 20:00:41.697996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.697924 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133"} err="failed to get container status \"ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133\": rpc error: code = NotFound desc = could not find container \"ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133\": container with ID starting with ebad4e0c134adeea51410456507f76bad6dd6dd7f9dceeac7664438b77e98133 not found: ID does not exist" Nov 02 20:00:41.894460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.894426 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49"] Nov 02 20:00:41.899468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:41.899440 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3acf2229-89c5bdb8c-rjt49"] Nov 02 20:00:42.003475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:42.003330 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" podStartSLOduration=2.003312463 podStartE2EDuration="2.003312463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:42.002462642 +0000 UTC m=+740.519235232" watchObservedRunningTime="2026-11-02 20:00:42.003312463 +0000 UTC m=+740.520085043" Nov 02 20:00:42.107191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:42.107159 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr"] Nov 02 20:00:42.109018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:42.108997 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3acf2229-b6f78675-f4srr"] Nov 02 20:00:42.347881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:42.347844 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp"] Nov 02 20:00:42.445558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:42.445520 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv"] Nov 02 20:00:43.620294 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:43.620255 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8cbaad-4073-4be8-8ff9-5551b6eca79f" path="/var/lib/kubelet/pods/0b8cbaad-4073-4be8-8ff9-5551b6eca79f/volumes" Nov 02 20:00:43.620675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:43.620666 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d780c0-9801-4f46-bd90-e0f102a877d7" path="/var/lib/kubelet/pods/82d780c0-9801-4f46-bd90-e0f102a877d7/volumes" Nov 02 20:00:46.381666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.381632 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv"] Nov 02 20:00:46.382112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.382070 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0b8cbaad-4073-4be8-8ff9-5551b6eca79f" containerName="payload-processing" Nov 02 20:00:46.382112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.382081 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8cbaad-4073-4be8-8ff9-5551b6eca79f" containerName="payload-processing" Nov 02 20:00:46.382112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.382104 5800 cpu_manager.go:414] "containerMap: removing container" podUID="82d780c0-9801-4f46-bd90-e0f102a877d7" containerName="payload-pre-processing" Nov 02 20:00:46.382112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.382109 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d780c0-9801-4f46-bd90-e0f102a877d7" containerName="payload-pre-processing" Nov 02 20:00:46.382260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.382189 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8cbaad-4073-4be8-8ff9-5551b6eca79f" containerName="payload-processing" Nov 02 20:00:46.382260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.382200 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d780c0-9801-4f46-bd90-e0f102a877d7" containerName="payload-pre-processing" Nov 02 20:00:46.386815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.386796 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.483024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.482993 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-14d090-dockercfg-2zbbr\"" Nov 02 20:00:46.483550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.483512 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-14d090\"" Nov 02 20:00:46.487488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.487461 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv"] Nov 02 20:00:46.514609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.514577 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ecd5b-ac98-4c72-8271-07b64797d0bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.514774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.514626 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzr4\" (UniqueName: \"kubernetes.io/projected/763ecd5b-ac98-4c72-8271-07b64797d0bf-kube-api-access-6pzr4\") pod \"payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.542375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.542344 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv"] Nov 02 20:00:46.546853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.546830 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.598364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.598330 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv"] Nov 02 20:00:46.615832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.615801 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ecd5b-ac98-4c72-8271-07b64797d0bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.616066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.615840 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4m6h\" (UniqueName: \"kubernetes.io/projected/662613f8-7642-4d75-86b9-f6fe2637d130-kube-api-access-t4m6h\") pod \"payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.616066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.615865 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzr4\" (UniqueName: \"kubernetes.io/projected/763ecd5b-ac98-4c72-8271-07b64797d0bf-kube-api-access-6pzr4\") pod \"payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.616066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.615885 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662613f8-7642-4d75-86b9-f6fe2637d130-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.616726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.616701 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ecd5b-ac98-4c72-8271-07b64797d0bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.669959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.669885 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2"] Nov 02 20:00:46.673768 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.673741 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.675712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.675684 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzr4\" (UniqueName: \"kubernetes.io/projected/763ecd5b-ac98-4c72-8271-07b64797d0bf-kube-api-access-6pzr4\") pod \"payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.700622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.700595 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750473 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750517 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rsm\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-kube-api-access-p5rsm\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750552 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4m6h\" (UniqueName: \"kubernetes.io/projected/662613f8-7642-4d75-86b9-f6fe2637d130-kube-api-access-t4m6h\") pod \"payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750593 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750632 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662613f8-7642-4d75-86b9-f6fe2637d130-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750670 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750719 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750750 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750781 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f18a58-c218-4d0c-98a1-cc05dcc53725-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750818 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.750841 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.751742 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2"] Nov 02 20:00:46.756304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.753152 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662613f8-7642-4d75-86b9-f6fe2637d130-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.794227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.794173 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c25349-openshift-default-dockercfg-5wzgl\"" Nov 02 20:00:46.806497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.806467 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4m6h\" (UniqueName: \"kubernetes.io/projected/662613f8-7642-4d75-86b9-f6fe2637d130-kube-api-access-t4m6h\") pod \"payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852148 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852206 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rsm\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-kube-api-access-p5rsm\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852262 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852344 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852401 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852434 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852463 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f18a58-c218-4d0c-98a1-cc05dcc53725-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852505 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.852528 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.854002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.853036 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.857587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.857387 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.857587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.857537 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.857780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.857750 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f18a58-c218-4d0c-98a1-cc05dcc53725-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.858106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.857994 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.858106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.858068 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.860161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.860141 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.862153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.862136 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:46.906791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.906764 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.907125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.907100 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rsm\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-kube-api-access-p5rsm\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:46.997249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:46.997209 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:47.135192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.135161 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv"] Nov 02 20:00:47.137169 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:47.137140 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763ecd5b_ac98_4c72_8271_07b64797d0bf.slice/crio-a3f3026a37f2f90763d5d168849ed91ade9feb9ac931da84b6dc1c8affe7da14 WatchSource:0}: Error finding container a3f3026a37f2f90763d5d168849ed91ade9feb9ac931da84b6dc1c8affe7da14: Status 404 returned error can't find the container with id a3f3026a37f2f90763d5d168849ed91ade9feb9ac931da84b6dc1c8affe7da14 Nov 02 20:00:47.240162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.240131 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv"] Nov 02 20:00:47.242356 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:47.242323 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662613f8_7642_4d75_86b9_f6fe2637d130.slice/crio-fe4de189e7bc3d28ca10e11821e6e2e1bd2716ad4c487e133648b960877501cc WatchSource:0}: Error finding container fe4de189e7bc3d28ca10e11821e6e2e1bd2716ad4c487e133648b960877501cc: Status 404 returned error can't find the container with id fe4de189e7bc3d28ca10e11821e6e2e1bd2716ad4c487e133648b960877501cc Nov 02 20:00:47.348350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.348314 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" podUID="a32704be-d93d-48c0-b184-1c63d92b8b28" containerName="payload-processing" containerID="cri-o://94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a" gracePeriod=25 Nov 02 20:00:47.371538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.371507 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2"] Nov 02 20:00:47.435501 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:47.435416 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f18a58_c218_4d0c_98a1_cc05dcc53725.slice/crio-fdc93bb04fb0637c45bec8c2fe94fe6d86c6c1e2f18106b0176d845be1d02e69 WatchSource:0}: Error finding container fdc93bb04fb0637c45bec8c2fe94fe6d86c6c1e2f18106b0176d845be1d02e69: Status 404 returned error can't find the container with id fdc93bb04fb0637c45bec8c2fe94fe6d86c6c1e2f18106b0176d845be1d02e69 Nov 02 20:00:47.446160 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.446122 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" podUID="2a31be42-1cbb-4599-bc04-4529d37f597a" containerName="payload-pre-processing" containerID="cri-o://1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc" gracePeriod=25 Nov 02 20:00:47.454460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.454422 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:47.454552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.454513 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:47.454590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.454554 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:00:47.595428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.595400 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 20:00:47.737724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.737696 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 20:00:47.761926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.761897 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32704be-d93d-48c0-b184-1c63d92b8b28-kube-api-access-mzjjg\" (UniqueName: \"kubernetes.io/projected/a32704be-d93d-48c0-b184-1c63d92b8b28-kube-api-access-mzjjg\") pod \"a32704be-d93d-48c0-b184-1c63d92b8b28\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " Nov 02 20:00:47.762086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.761932 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32704be-d93d-48c0-b184-1c63d92b8b28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32704be-d93d-48c0-b184-1c63d92b8b28-plugins-config-volume\") pod \"a32704be-d93d-48c0-b184-1c63d92b8b28\" (UID: \"a32704be-d93d-48c0-b184-1c63d92b8b28\") " Nov 02 20:00:47.763179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.762950 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32704be-d93d-48c0-b184-1c63d92b8b28-plugins-config-volume" pod "a32704be-d93d-48c0-b184-1c63d92b8b28" (UID: "a32704be-d93d-48c0-b184-1c63d92b8b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:47.766352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.764486 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32704be-d93d-48c0-b184-1c63d92b8b28-kube-api-access-mzjjg" pod "a32704be-d93d-48c0-b184-1c63d92b8b28" (UID: "a32704be-d93d-48c0-b184-1c63d92b8b28"). InnerVolumeSpecName "kube-api-access-mzjjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:47.779471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.779437 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" event={"ID":"763ecd5b-ac98-4c72-8271-07b64797d0bf","Type":"ContainerStarted","Data":"f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1"} Nov 02 20:00:47.779610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.779539 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:47.779610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.779566 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" event={"ID":"763ecd5b-ac98-4c72-8271-07b64797d0bf","Type":"ContainerStarted","Data":"a3f3026a37f2f90763d5d168849ed91ade9feb9ac931da84b6dc1c8affe7da14"} Nov 02 20:00:47.781314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.781265 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" event={"ID":"662613f8-7642-4d75-86b9-f6fe2637d130","Type":"ContainerStarted","Data":"735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4"} Nov 02 20:00:47.781504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.781484 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:47.781589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.781520 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" event={"ID":"662613f8-7642-4d75-86b9-f6fe2637d130","Type":"ContainerStarted","Data":"fe4de189e7bc3d28ca10e11821e6e2e1bd2716ad4c487e133648b960877501cc"} Nov 02 20:00:47.782720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.782693 5800 generic.go:356] "Generic (PLEG): container finished" podID="2a31be42-1cbb-4599-bc04-4529d37f597a" containerID="1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc" exitCode=0 Nov 02 20:00:47.782807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.782741 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" Nov 02 20:00:47.782807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.782774 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" event={"ID":"2a31be42-1cbb-4599-bc04-4529d37f597a","Type":"ContainerDied","Data":"1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc"} Nov 02 20:00:47.782895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.782825 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv" event={"ID":"2a31be42-1cbb-4599-bc04-4529d37f597a","Type":"ContainerDied","Data":"0972170cd62697c4d3a6454bd252baef5389b67c8cde708516387202f198a0d7"} Nov 02 20:00:47.782895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.782850 5800 scope.go:122] "RemoveContainer" containerID="1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc" Nov 02 20:00:47.784192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.784063 5800 generic.go:356] "Generic (PLEG): container finished" podID="a32704be-d93d-48c0-b184-1c63d92b8b28" containerID="94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a" exitCode=0 Nov 02 20:00:47.784192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.784127 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" Nov 02 20:00:47.784192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.784135 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" event={"ID":"a32704be-d93d-48c0-b184-1c63d92b8b28","Type":"ContainerDied","Data":"94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a"} Nov 02 20:00:47.784192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.784183 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp" event={"ID":"a32704be-d93d-48c0-b184-1c63d92b8b28","Type":"ContainerDied","Data":"dd0ac07574150c1ea8bc34bdc319a5b120bf5b674eb7753a728b9736ba78c1ef"} Nov 02 20:00:47.785965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.785940 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" event={"ID":"97f18a58-c218-4d0c-98a1-cc05dcc53725","Type":"ContainerStarted","Data":"7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf"} Nov 02 20:00:47.786078 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.785990 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" event={"ID":"97f18a58-c218-4d0c-98a1-cc05dcc53725","Type":"ContainerStarted","Data":"fdc93bb04fb0637c45bec8c2fe94fe6d86c6c1e2f18106b0176d845be1d02e69"} Nov 02 20:00:47.794836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.794819 5800 scope.go:122] "RemoveContainer" containerID="1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc" Nov 02 20:00:47.795225 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:47.795134 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc\": container with ID starting with 1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc not found: ID does not exist" containerID="1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc" Nov 02 20:00:47.795225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.795169 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc"} err="failed to get container status \"1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc\": rpc error: code = NotFound desc = could not find container \"1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc\": container with ID starting with 1d347d0b0e6194f34f6ab2795ccc5c54942659e3392e508c27c604245edbf6bc not found: ID does not exist" Nov 02 20:00:47.795225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.795191 5800 scope.go:122] "RemoveContainer" containerID="94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a" Nov 02 20:00:47.808439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.808416 5800 scope.go:122] "RemoveContainer" containerID="94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a" Nov 02 20:00:47.808754 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:47.808737 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a\": container with ID starting with 94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a not found: ID does not exist" containerID="94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a" Nov 02 20:00:47.808828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.808767 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a"} err="failed to get container status \"94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a\": rpc error: code = NotFound desc = could not find container \"94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a\": container with ID starting with 94a4c2bc90e08dda152bb12b2013f2996261ff438c6ad0eb591094467529113a not found: ID does not exist" Nov 02 20:00:47.865080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.865045 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a31be42-1cbb-4599-bc04-4529d37f597a-kube-api-access-kzl2g\" (UniqueName: \"kubernetes.io/projected/2a31be42-1cbb-4599-bc04-4529d37f597a-kube-api-access-kzl2g\") pod \"2a31be42-1cbb-4599-bc04-4529d37f597a\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " Nov 02 20:00:47.865274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.865154 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a31be42-1cbb-4599-bc04-4529d37f597a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a31be42-1cbb-4599-bc04-4529d37f597a-plugins-config-volume\") pod \"2a31be42-1cbb-4599-bc04-4529d37f597a\" (UID: \"2a31be42-1cbb-4599-bc04-4529d37f597a\") " Nov 02 20:00:47.865617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.865586 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a31be42-1cbb-4599-bc04-4529d37f597a-plugins-config-volume" pod "2a31be42-1cbb-4599-bc04-4529d37f597a" (UID: "2a31be42-1cbb-4599-bc04-4529d37f597a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:00:47.865731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.865616 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjjg\" (UniqueName: \"kubernetes.io/projected/a32704be-d93d-48c0-b184-1c63d92b8b28-kube-api-access-mzjjg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:47.865731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.865640 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a32704be-d93d-48c0-b184-1c63d92b8b28-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:47.868091 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.868060 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a31be42-1cbb-4599-bc04-4529d37f597a-kube-api-access-kzl2g" pod "2a31be42-1cbb-4599-bc04-4529d37f597a" (UID: "2a31be42-1cbb-4599-bc04-4529d37f597a"). InnerVolumeSpecName "kube-api-access-kzl2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:00:47.952484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.952385 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" podStartSLOduration=1.9523697759999998 podStartE2EDuration="1.952369776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:47.950407524 +0000 UTC m=+746.467180111" watchObservedRunningTime="2026-11-02 20:00:47.952369776 +0000 UTC m=+746.469142362" Nov 02 20:00:47.966948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.966916 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzl2g\" (UniqueName: \"kubernetes.io/projected/2a31be42-1cbb-4599-bc04-4529d37f597a-kube-api-access-kzl2g\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:47.966948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.966946 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a31be42-1cbb-4599-bc04-4529d37f597a-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:00:47.997797 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:47.997750 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:48.059183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:48.059126 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" podStartSLOduration=2.059112915 podStartE2EDuration="2.059112915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:48.057169212 +0000 UTC m=+746.573941788" watchObservedRunningTime="2026-11-02 20:00:48.059112915 +0000 UTC m=+746.575885525" Nov 02 20:00:48.164085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:48.164049 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp"] Nov 02 20:00:48.167432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:48.167403 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-172adb07-86c96dccf-vfxfp"] Nov 02 20:00:48.442039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:48.438888 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podStartSLOduration=2.438866109 podStartE2EDuration="2.438866109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:48.334819669 +0000 UTC m=+746.851592257" watchObservedRunningTime="2026-11-02 20:00:48.438866109 +0000 UTC m=+746.955638695" Nov 02 20:00:48.442039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:48.439905 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv"] Nov 02 20:00:48.443347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:48.443265 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-172adb07-8d7f948dc-rrclv"] Nov 02 20:00:49.003863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:49.003827 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:49.621453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:49.621419 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a31be42-1cbb-4599-bc04-4529d37f597a" path="/var/lib/kubelet/pods/2a31be42-1cbb-4599-bc04-4529d37f597a/volumes" Nov 02 20:00:49.621853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:49.621798 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32704be-d93d-48c0-b184-1c63d92b8b28" path="/var/lib/kubelet/pods/a32704be-d93d-48c0-b184-1c63d92b8b28/volumes" Nov 02 20:00:49.799928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:49.799896 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:49.800969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:49.800947 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:00:50.935624 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.935590 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st"] Nov 02 20:00:50.936041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.936028 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2a31be42-1cbb-4599-bc04-4529d37f597a" containerName="payload-pre-processing" Nov 02 20:00:50.936105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.936041 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a31be42-1cbb-4599-bc04-4529d37f597a" containerName="payload-pre-processing" Nov 02 20:00:50.936105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.936067 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a32704be-d93d-48c0-b184-1c63d92b8b28" containerName="payload-processing" Nov 02 20:00:50.936105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.936073 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32704be-d93d-48c0-b184-1c63d92b8b28" containerName="payload-processing" Nov 02 20:00:50.936195 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.936142 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32704be-d93d-48c0-b184-1c63d92b8b28" containerName="payload-processing" Nov 02 20:00:50.936195 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.936153 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a31be42-1cbb-4599-bc04-4529d37f597a" containerName="payload-pre-processing" Nov 02 20:00:50.940960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.940935 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:50.991703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:50.991666 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st"] Nov 02 20:00:51.035065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.035032 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6a4ab0\"" Nov 02 20:00:51.035065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.035052 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-6a4ab0-dockercfg-sw8b7\"" Nov 02 20:00:51.112109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.112067 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190971c5-38b5-4649-98b2-fe1b4b9390c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:51.112587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.112561 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjh7\" (UniqueName: \"kubernetes.io/projected/190971c5-38b5-4649-98b2-fe1b4b9390c8-kube-api-access-mtjh7\") pod \"payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:51.112887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.112869 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8"] Nov 02 20:00:51.122276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.122245 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.163976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.163939 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8"] Nov 02 20:00:51.213560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.213470 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjh7\" (UniqueName: \"kubernetes.io/projected/190971c5-38b5-4649-98b2-fe1b4b9390c8-kube-api-access-mtjh7\") pod \"payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:51.213560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.213519 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t875\" (UniqueName: \"kubernetes.io/projected/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-kube-api-access-4t875\") pod \"payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.213763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.213588 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190971c5-38b5-4649-98b2-fe1b4b9390c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:51.213763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.213726 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.214354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.214335 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190971c5-38b5-4649-98b2-fe1b4b9390c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:51.266175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.266136 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjh7\" (UniqueName: \"kubernetes.io/projected/190971c5-38b5-4649-98b2-fe1b4b9390c8-kube-api-access-mtjh7\") pod \"payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:51.315372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.315331 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.315531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.315437 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t875\" (UniqueName: \"kubernetes.io/projected/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-kube-api-access-4t875\") pod \"payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.316158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.316122 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.368190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.368156 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t875\" (UniqueName: \"kubernetes.io/projected/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-kube-api-access-4t875\") pod \"payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.441906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.441866 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:51.557237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.557194 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:51.807851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.807817 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8"] Nov 02 20:00:51.809222 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:51.809198 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73e60ab_304a_4bba_93c8_7ae5ab8e7715.slice/crio-67124d2d1fecc59c73d38c8377c0f56ccc9cbe85e3b159a9dd61af63406c1a04 WatchSource:0}: Error finding container 67124d2d1fecc59c73d38c8377c0f56ccc9cbe85e3b159a9dd61af63406c1a04: Status 404 returned error can't find the container with id 67124d2d1fecc59c73d38c8377c0f56ccc9cbe85e3b159a9dd61af63406c1a04 Nov 02 20:00:51.924595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:51.924570 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st"] Nov 02 20:00:51.926637 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:00:51.926602 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190971c5_38b5_4649_98b2_fe1b4b9390c8.slice/crio-931355431dd3ea7327035844565467776ca8b7328afd604adb41c9da52113012 WatchSource:0}: Error finding container 931355431dd3ea7327035844565467776ca8b7328afd604adb41c9da52113012: Status 404 returned error can't find the container with id 931355431dd3ea7327035844565467776ca8b7328afd604adb41c9da52113012 Nov 02 20:00:52.693696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.693664 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 20:00:52.697069 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:00:52.697036 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:00:52.820723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.820681 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" event={"ID":"190971c5-38b5-4649-98b2-fe1b4b9390c8","Type":"ContainerStarted","Data":"3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c"} Nov 02 20:00:52.820925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.820861 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:00:52.820925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.820915 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" event={"ID":"190971c5-38b5-4649-98b2-fe1b4b9390c8","Type":"ContainerStarted","Data":"931355431dd3ea7327035844565467776ca8b7328afd604adb41c9da52113012"} Nov 02 20:00:52.822323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.822303 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" event={"ID":"a73e60ab-304a-4bba-93c8-7ae5ab8e7715","Type":"ContainerStarted","Data":"c3a4df3e095e39713f0187f497eee05ba37f261a5e365692f525c8056901cff3"} Nov 02 20:00:52.822432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.822338 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" event={"ID":"a73e60ab-304a-4bba-93c8-7ae5ab8e7715","Type":"ContainerStarted","Data":"67124d2d1fecc59c73d38c8377c0f56ccc9cbe85e3b159a9dd61af63406c1a04"} Nov 02 20:00:52.822432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.822359 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:00:52.932118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:52.932061 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" podStartSLOduration=2.932046173 podStartE2EDuration="2.932046173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:52.931357886 +0000 UTC m=+751.448130470" watchObservedRunningTime="2026-11-02 20:00:52.932046173 +0000 UTC m=+751.448818755" Nov 02 20:00:53.039943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:53.039880 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" podStartSLOduration=2.039866997 podStartE2EDuration="2.039866997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:00:53.036887289 +0000 UTC m=+751.553659877" watchObservedRunningTime="2026-11-02 20:00:53.039866997 +0000 UTC m=+751.556639581" Nov 02 20:00:58.798010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:58.797979 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 20:00:58.799190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:58.799167 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 20:00:59.904467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:59.904431 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d"] Nov 02 20:00:59.909762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:59.909739 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:00:59.962233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:00:59.962193 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d"] Nov 02 20:01:00.005682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.005156 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c25349-dockercfg-4zbjg\"" Nov 02 20:01:00.005682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.005312 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c25349\"" Nov 02 20:01:00.008127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.007804 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28m5m\" (UniqueName: \"kubernetes.io/projected/cd36395c-5873-43a8-af0a-ab83c08ab4b0-kube-api-access-28m5m\") pod \"payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.008336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.008226 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd36395c-5873-43a8-af0a-ab83c08ab4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.066871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.066836 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g"] Nov 02 20:01:00.070655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.070638 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.109804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.109769 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd36395c-5873-43a8-af0a-ab83c08ab4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.109981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.109842 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00724ebe-42d2-43b5-b8a9-86c172611d38-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.109981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.109881 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28m5m\" (UniqueName: \"kubernetes.io/projected/cd36395c-5873-43a8-af0a-ab83c08ab4b0-kube-api-access-28m5m\") pod \"payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.109981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.109901 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jn9\" (UniqueName: \"kubernetes.io/projected/00724ebe-42d2-43b5-b8a9-86c172611d38-kube-api-access-p5jn9\") pod \"payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.110725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.110705 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd36395c-5873-43a8-af0a-ab83c08ab4b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.124964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.124934 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g"] Nov 02 20:01:00.163571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.163501 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28m5m\" (UniqueName: \"kubernetes.io/projected/cd36395c-5873-43a8-af0a-ab83c08ab4b0-kube-api-access-28m5m\") pod \"payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.211380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.211344 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00724ebe-42d2-43b5-b8a9-86c172611d38-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.211598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.211404 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jn9\" (UniqueName: \"kubernetes.io/projected/00724ebe-42d2-43b5-b8a9-86c172611d38-kube-api-access-p5jn9\") pod \"payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.212087 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.212067 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00724ebe-42d2-43b5-b8a9-86c172611d38-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.223259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.223239 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.264714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.264675 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jn9\" (UniqueName: \"kubernetes.io/projected/00724ebe-42d2-43b5-b8a9-86c172611d38-kube-api-access-p5jn9\") pod \"payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.385659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.384637 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:00.618149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.618126 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d"] Nov 02 20:01:00.619349 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:01:00.619324 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd36395c_5873_43a8_af0a_ab83c08ab4b0.slice/crio-304fab7e6a1a95e86f2f9d548bba3f829d85b705f28f09a8896c246dc1ca4652 WatchSource:0}: Error finding container 304fab7e6a1a95e86f2f9d548bba3f829d85b705f28f09a8896c246dc1ca4652: Status 404 returned error can't find the container with id 304fab7e6a1a95e86f2f9d548bba3f829d85b705f28f09a8896c246dc1ca4652 Nov 02 20:01:00.781011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.780980 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g"] Nov 02 20:01:00.783387 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:01:00.783352 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00724ebe_42d2_43b5_b8a9_86c172611d38.slice/crio-35b808f4436841b2a7ef81bc1ba3513ca72e06fa6548b979d85c98d61f3427d4 WatchSource:0}: Error finding container 35b808f4436841b2a7ef81bc1ba3513ca72e06fa6548b979d85c98d61f3427d4: Status 404 returned error can't find the container with id 35b808f4436841b2a7ef81bc1ba3513ca72e06fa6548b979d85c98d61f3427d4 Nov 02 20:01:00.887940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.887904 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" event={"ID":"cd36395c-5873-43a8-af0a-ab83c08ab4b0","Type":"ContainerStarted","Data":"8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33"} Nov 02 20:01:00.888081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.887976 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:00.888081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.888002 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" event={"ID":"cd36395c-5873-43a8-af0a-ab83c08ab4b0","Type":"ContainerStarted","Data":"304fab7e6a1a95e86f2f9d548bba3f829d85b705f28f09a8896c246dc1ca4652"} Nov 02 20:01:00.889860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.889836 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" event={"ID":"00724ebe-42d2-43b5-b8a9-86c172611d38","Type":"ContainerStarted","Data":"49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596"} Nov 02 20:01:00.889995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.889879 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" event={"ID":"00724ebe-42d2-43b5-b8a9-86c172611d38","Type":"ContainerStarted","Data":"35b808f4436841b2a7ef81bc1ba3513ca72e06fa6548b979d85c98d61f3427d4"} Nov 02 20:01:00.889995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:00.889921 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:01.004683 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:01.004631 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" podStartSLOduration=2.004617735 podStartE2EDuration="2.004617735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:01.002380916 +0000 UTC m=+759.519153502" watchObservedRunningTime="2026-11-02 20:01:01.004617735 +0000 UTC m=+759.521390319" Nov 02 20:01:01.111231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:01.111121 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" podStartSLOduration=1.111106683 podStartE2EDuration="1.111106683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:01:01.109670272 +0000 UTC m=+759.626442857" watchObservedRunningTime="2026-11-02 20:01:01.111106683 +0000 UTC m=+759.627879267" Nov 02 20:01:03.652429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:03.652388 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:01:03.652895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:03.652452 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:01:03.831668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:03.831633 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 20:01:03.831864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:03.831718 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 20:01:05.922405 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:05.922361 5800 generic.go:356] "Generic (PLEG): container finished" podID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerID="815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810" exitCode=0 Nov 02 20:01:05.922958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:05.922403 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" event={"ID":"b8398ad9-e519-4c49-a40e-1efedaf04d5d","Type":"ContainerDied","Data":"815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810"} Nov 02 20:01:05.922958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:05.922804 5800 scope.go:122] "RemoveContainer" containerID="815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810" Nov 02 20:01:05.924167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:05.924142 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:05.924232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:05.924209 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:05.924275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:05.924240 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:07.007716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:07.007687 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" event={"ID":"b8398ad9-e519-4c49-a40e-1efedaf04d5d","Type":"ContainerStarted","Data":"03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4"} Nov 02 20:01:07.330735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:07.330646 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:01:07.331088 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:07.330995 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:01:07.331302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:07.331253 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 20:01:08.333307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:08.329744 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:01:08.333307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:08.329820 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 20:01:09.329782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:09.329738 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:01:09.330121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:09.330088 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 20:01:10.330822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:10.330786 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:01:10.331464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:10.331417 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 20:01:11.331308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:11.329664 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 20:01:11.331308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:11.329726 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 20:01:11.899357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:11.899311 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 20:01:11.899467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:11.899392 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 20:01:12.331365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:12.331335 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:01:12.331811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:12.331787 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:01:12.332624 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:12.332601 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:01:16.123004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:16.122966 5800 generic.go:356] "Generic (PLEG): container finished" podID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerID="e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12" exitCode=0 Nov 02 20:01:16.123521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:16.123005 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" event={"ID":"3de6c98b-c7c5-452c-ba6c-87179d8a5127","Type":"ContainerDied","Data":"e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12"} Nov 02 20:01:16.123571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:16.123530 5800 scope.go:122] "RemoveContainer" containerID="e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12" Nov 02 20:01:16.124976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:16.124948 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:16.125071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:16.125017 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:16.125071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:16.125052 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:17.153313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:17.153245 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" event={"ID":"3de6c98b-c7c5-452c-ba6c-87179d8a5127","Type":"ContainerStarted","Data":"e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427"} Nov 02 20:01:18.124182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:18.124141 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:01:18.139566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:18.139523 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:01:18.140112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:18.139910 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 20:01:19.123868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:19.123822 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:01:19.124400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:19.123916 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 20:01:19.638305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:19.629369 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:01:20.124050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:20.124008 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:01:20.124516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:20.124073 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 20:01:21.124606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:21.124574 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:01:21.125129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:21.124625 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 20:01:22.124760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:22.124720 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 20:01:22.125157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:22.124778 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 20:01:22.696331 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:01:22.696262 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:01:23.128613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:23.128579 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:01:23.129815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:23.129790 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:01:33.274529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:33.274493 5800 generic.go:356] "Generic (PLEG): container finished" podID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerID="7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf" exitCode=0 Nov 02 20:01:33.274952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:33.274562 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" event={"ID":"97f18a58-c218-4d0c-98a1-cc05dcc53725","Type":"ContainerDied","Data":"7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf"} Nov 02 20:01:33.274952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:33.274930 5800 scope.go:122] "RemoveContainer" containerID="7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf" Nov 02 20:01:33.276234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:33.276210 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:33.276314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:33.276271 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:33.276360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:33.276321 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:01:34.282945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:34.282910 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" event={"ID":"97f18a58-c218-4d0c-98a1-cc05dcc53725","Type":"ContainerStarted","Data":"c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4"} Nov 02 20:01:34.800857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:34.800813 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:01:34.997584 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:34.997552 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:01:34.998722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:34.998698 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:01:34.998820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:34.998744 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 20:01:35.998115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:35.998073 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:01:35.998601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:35.998141 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 20:01:36.997854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:36.997817 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:01:36.998074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:36.997918 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 20:01:37.998355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:37.998317 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:01:37.998775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:37.998374 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 20:01:38.998184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:38.998146 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 20:01:38.998392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:38.998207 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 20:01:40.002080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:40.002047 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:01:40.003204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:01:40.003185 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:01:52.700311 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:01:52.700224 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:02:07.148822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.148740 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj"] Nov 02 20:02:07.152930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.152909 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.207059 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.206981 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj"] Nov 02 20:02:07.247101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.247075 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-tmp-dir\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.247226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.247108 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0435af2-55cd-4cc3-b129-22a290db98ed-tls-certs\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.247309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.247262 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpq44\" (UniqueName: \"kubernetes.io/projected/c0435af2-55cd-4cc3-b129-22a290db98ed-kube-api-access-rpq44\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.247364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.247352 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-model-cache\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.247418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.247405 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-home\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.247458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.247449 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-dshm\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.249275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.249255 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"kube-root-ca.crt\"" Nov 02 20:02:07.249666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.249651 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"openshift-service-ca.crt\"" Nov 02 20:02:07.249710 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.249649 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"default-dockercfg-hxc6x\"" Nov 02 20:02:07.249813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.249797 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"premium-w2-426941-kserve-self-signed-certs\"" Nov 02 20:02:07.348780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.348727 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-dshm\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.348985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.348788 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-tmp-dir\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.348985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.348814 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0435af2-55cd-4cc3-b129-22a290db98ed-tls-certs\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.348985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.348899 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpq44\" (UniqueName: \"kubernetes.io/projected/c0435af2-55cd-4cc3-b129-22a290db98ed-kube-api-access-rpq44\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.348985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.348943 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-model-cache\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.349217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.349011 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-home\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.349277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.349242 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-tmp-dir\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.349394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.349375 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-home\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.349445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.349413 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-model-cache\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.351129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.351109 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-dshm\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.351465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.351444 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0435af2-55cd-4cc3-b129-22a290db98ed-tls-certs\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.402333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.402260 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpq44\" (UniqueName: \"kubernetes.io/projected/c0435af2-55cd-4cc3-b129-22a290db98ed-kube-api-access-rpq44\") pod \"premium-w2-426941-kserve-6b7dbb77cb-vhdgj\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.490711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.490679 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:07.870051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:07.870023 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj"] Nov 02 20:02:07.871492 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:02:07.871470 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0435af2_55cd_4cc3_b129_22a290db98ed.slice/crio-44689b0c4f12507660b5394b9d5fd817ebba6cfb0c8eec900012e808a9d5ec9c WatchSource:0}: Error finding container 44689b0c4f12507660b5394b9d5fd817ebba6cfb0c8eec900012e808a9d5ec9c: Status 404 returned error can't find the container with id 44689b0c4f12507660b5394b9d5fd817ebba6cfb0c8eec900012e808a9d5ec9c Nov 02 20:02:08.496469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:08.496429 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" event={"ID":"c0435af2-55cd-4cc3-b129-22a290db98ed","Type":"ContainerStarted","Data":"2bca9f0668540e5ada85a82313071845b2fdeb71d48e55dedcaef119a166fca4"} Nov 02 20:02:08.496469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:08.496476 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" event={"ID":"c0435af2-55cd-4cc3-b129-22a290db98ed","Type":"ContainerStarted","Data":"44689b0c4f12507660b5394b9d5fd817ebba6cfb0c8eec900012e808a9d5ec9c"} Nov 02 20:02:08.604204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:08.604151 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podStartSLOduration=1.604136822 podStartE2EDuration="1.604136822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:08.603350621 +0000 UTC m=+827.120123208" watchObservedRunningTime="2026-11-02 20:02:08.604136822 +0000 UTC m=+827.120909406" Nov 02 20:02:09.491020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:09.490980 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:17.491245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:17.491207 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:17.504918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:17.504885 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:17.564764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:17.564738 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:02:22.695789 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:02:22.695754 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:02:40.397705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.397670 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc"] Nov 02 20:02:40.401805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.401782 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.453157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.453119 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"kube-root-ca.crt\"" Nov 02 20:02:40.498361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.498323 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"openshift-service-ca.crt\"" Nov 02 20:02:40.498861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.498840 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"premium-w0-14d090-kserve-self-signed-certs\"" Nov 02 20:02:40.498974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.498839 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-14d090\"/\"default-dockercfg-xfwx7\"" Nov 02 20:02:40.503889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.503860 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc"] Nov 02 20:02:40.509668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.508300 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-model-cache\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.509668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.509355 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5af302-63fa-46da-883a-740f4c486be5-tls-certs\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.509668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.509451 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-home\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.509668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.509506 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-tmp-dir\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.509668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.509565 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq74r\" (UniqueName: \"kubernetes.io/projected/ac5af302-63fa-46da-883a-740f4c486be5-kube-api-access-sq74r\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.510347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.510315 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-dshm\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.611485 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.611447 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-dshm\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.611661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.611512 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-model-cache\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.611661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.611538 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5af302-63fa-46da-883a-740f4c486be5-tls-certs\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.611661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.611625 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-home\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.611778 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.611666 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-tmp-dir\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.611778 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.611711 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq74r\" (UniqueName: \"kubernetes.io/projected/ac5af302-63fa-46da-883a-740f4c486be5-kube-api-access-sq74r\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.611959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.611931 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-model-cache\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.612186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.612162 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-home\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.612353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.612324 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-tmp-dir\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.614522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.614500 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5af302-63fa-46da-883a-740f4c486be5-tls-certs\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.617523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.617496 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-dshm\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.668148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.668073 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq74r\" (UniqueName: \"kubernetes.io/projected/ac5af302-63fa-46da-883a-740f4c486be5-kube-api-access-sq74r\") pod \"premium-w0-14d090-kserve-59869d8966-2dmrc\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:40.714403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:40.714373 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:41.092148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:41.092121 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc"] Nov 02 20:02:41.093769 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:02:41.093745 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac5af302_63fa_46da_883a_740f4c486be5.slice/crio-74d0ec3680ebf0d5568bb837bc34748e3550453edc3ec08931041a846158e0fd WatchSource:0}: Error finding container 74d0ec3680ebf0d5568bb837bc34748e3550453edc3ec08931041a846158e0fd: Status 404 returned error can't find the container with id 74d0ec3680ebf0d5568bb837bc34748e3550453edc3ec08931041a846158e0fd Nov 02 20:02:41.702638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:41.702605 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" event={"ID":"ac5af302-63fa-46da-883a-740f4c486be5","Type":"ContainerStarted","Data":"dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e"} Nov 02 20:02:41.702638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:41.702646 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" event={"ID":"ac5af302-63fa-46da-883a-740f4c486be5","Type":"ContainerStarted","Data":"74d0ec3680ebf0d5568bb837bc34748e3550453edc3ec08931041a846158e0fd"} Nov 02 20:02:41.714727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:41.714703 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:41.811308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:41.811231 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podStartSLOduration=1.8112159650000001 podStartE2EDuration="1.811215965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:02:41.809080475 +0000 UTC m=+860.325853063" watchObservedRunningTime="2026-11-02 20:02:41.811215965 +0000 UTC m=+860.327988549" Nov 02 20:02:50.715275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:50.715241 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:50.728031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:50.728005 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:50.812401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:02:50.812373 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:02:52.696215 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:02:52.696182 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:03:21.743651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:21.743621 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:03:21.745002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:21.744983 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:03:22.696953 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:03:22.696909 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:03:23.050986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.050951 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98"] Nov 02 20:03:23.055248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.055232 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.107263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.107234 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98"] Nov 02 20:03:23.150260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.150212 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:03:23.150260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.150222 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:03:23.150525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.150309 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:03:23.150923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.150906 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:03:23.180575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.180545 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgq7\" (UniqueName: \"kubernetes.io/projected/89d871ed-04f2-4fff-bda8-016983c49e15-kube-api-access-jrgq7\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.180744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.180597 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.180744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.180717 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.281567 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.281532 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgq7\" (UniqueName: \"kubernetes.io/projected/89d871ed-04f2-4fff-bda8-016983c49e15-kube-api-access-jrgq7\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.281751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.281596 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.281751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.281678 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.282369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.282339 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.284011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.283993 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.335897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.335808 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgq7\" (UniqueName: \"kubernetes.io/projected/89d871ed-04f2-4fff-bda8-016983c49e15-kube-api-access-jrgq7\") pod \"maas-api-revoke-sub-c8e53c24-5kn98\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.368576 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.368541 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:23.737216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:23.737187 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98"] Nov 02 20:03:23.739354 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:03:23.739321 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d871ed_04f2_4fff_bda8_016983c49e15.slice/crio-26078d495e777865050c76af18383ac04fca5ab7e2bdb3ee06dbb969b6a41b1a WatchSource:0}: Error finding container 26078d495e777865050c76af18383ac04fca5ab7e2bdb3ee06dbb969b6a41b1a: Status 404 returned error can't find the container with id 26078d495e777865050c76af18383ac04fca5ab7e2bdb3ee06dbb969b6a41b1a Nov 02 20:03:24.010021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:24.009989 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" event={"ID":"89d871ed-04f2-4fff-bda8-016983c49e15","Type":"ContainerStarted","Data":"26078d495e777865050c76af18383ac04fca5ab7e2bdb3ee06dbb969b6a41b1a"} Nov 02 20:03:25.017628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:25.017591 5800 generic.go:356] "Generic (PLEG): container finished" podID="89d871ed-04f2-4fff-bda8-016983c49e15" containerID="1a31497d2d0beb7dadd602b6dda2c75f6e0db884fa3dfa20e4c6cb5aa50b3e68" exitCode=0 Nov 02 20:03:25.018053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:25.017680 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" event={"ID":"89d871ed-04f2-4fff-bda8-016983c49e15","Type":"ContainerDied","Data":"1a31497d2d0beb7dadd602b6dda2c75f6e0db884fa3dfa20e4c6cb5aa50b3e68"} Nov 02 20:03:26.190678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.190649 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:26.325629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.325538 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d871ed-04f2-4fff-bda8-016983c49e15-kube-api-access-jrgq7\" (UniqueName: \"kubernetes.io/projected/89d871ed-04f2-4fff-bda8-016983c49e15-kube-api-access-jrgq7\") pod \"89d871ed-04f2-4fff-bda8-016983c49e15\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " Nov 02 20:03:26.325629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.325601 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-xks-ca\") pod \"89d871ed-04f2-4fff-bda8-016983c49e15\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " Nov 02 20:03:26.325629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.325623 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-openshift-ca\") pod \"89d871ed-04f2-4fff-bda8-016983c49e15\" (UID: \"89d871ed-04f2-4fff-bda8-016983c49e15\") " Nov 02 20:03:26.326093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.326055 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-openshift-ca" pod "89d871ed-04f2-4fff-bda8-016983c49e15" (UID: "89d871ed-04f2-4fff-bda8-016983c49e15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:26.327794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.327766 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d871ed-04f2-4fff-bda8-016983c49e15-kube-api-access-jrgq7" pod "89d871ed-04f2-4fff-bda8-016983c49e15" (UID: "89d871ed-04f2-4fff-bda8-016983c49e15"). InnerVolumeSpecName "kube-api-access-jrgq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:26.327892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.327810 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-xks-ca" pod "89d871ed-04f2-4fff-bda8-016983c49e15" (UID: "89d871ed-04f2-4fff-bda8-016983c49e15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:26.427570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.427529 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrgq7\" (UniqueName: \"kubernetes.io/projected/89d871ed-04f2-4fff-bda8-016983c49e15-kube-api-access-jrgq7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:26.427570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.427563 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:26.427570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:26.427574 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89d871ed-04f2-4fff-bda8-016983c49e15-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:27.033697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:27.033658 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" event={"ID":"89d871ed-04f2-4fff-bda8-016983c49e15","Type":"ContainerDied","Data":"26078d495e777865050c76af18383ac04fca5ab7e2bdb3ee06dbb969b6a41b1a"} Nov 02 20:03:27.034408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:27.033713 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26078d495e777865050c76af18383ac04fca5ab7e2bdb3ee06dbb969b6a41b1a" Nov 02 20:03:27.034408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:27.033675 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98" Nov 02 20:03:34.883746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:34.883661 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z"] Nov 02 20:03:34.884195 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:34.884182 5800 cpu_manager.go:414] "containerMap: removing container" podUID="89d871ed-04f2-4fff-bda8-016983c49e15" containerName="revoke-keys" Nov 02 20:03:34.884242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:34.884197 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d871ed-04f2-4fff-bda8-016983c49e15" containerName="revoke-keys" Nov 02 20:03:34.884320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:34.884310 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d871ed-04f2-4fff-bda8-016983c49e15" containerName="revoke-keys" Nov 02 20:03:34.887610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:34.887592 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:34.936723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:34.936693 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a4ab0-openshift-default-dockercfg-xzj7t\"" Nov 02 20:03:34.985969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:34.985931 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z"] Nov 02 20:03:35.017020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.016982 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017035 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017155 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-kube-api-access-sjrtb\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017197 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017355 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017424 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017484 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017541 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.017598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.017587 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118428 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118471 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118601 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118644 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118711 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-kube-api-access-sjrtb\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118758 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118825 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118853 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.118902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118892 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.119005 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118930 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.119005 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118943 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.119005 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.118991 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.119152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.119013 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.119558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.119536 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.121072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.121045 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.121275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.121256 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.172235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.172172 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.172366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.172348 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-kube-api-access-sjrtb\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.201014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.200985 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:35.616917 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:03:35.616881 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbf9e3f_d603_4c75_ae01_f9c277be7676.slice/crio-9f27199487eefb321ca64110dc983faf9d60835067cdd6713ca3027e2bd95853 WatchSource:0}: Error finding container 9f27199487eefb321ca64110dc983faf9d60835067cdd6713ca3027e2bd95853: Status 404 returned error can't find the container with id 9f27199487eefb321ca64110dc983faf9d60835067cdd6713ca3027e2bd95853 Nov 02 20:03:35.619612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.619571 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:35.619706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.619659 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:35.619706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.619697 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:03:35.625634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:35.625611 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z"] Nov 02 20:03:36.104181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:36.104138 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" event={"ID":"4fbf9e3f-d603-4c75-ae01-f9c277be7676","Type":"ContainerStarted","Data":"c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734"} Nov 02 20:03:36.104674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:36.104243 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" event={"ID":"4fbf9e3f-d603-4c75-ae01-f9c277be7676","Type":"ContainerStarted","Data":"9f27199487eefb321ca64110dc983faf9d60835067cdd6713ca3027e2bd95853"} Nov 02 20:03:36.201781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:36.201743 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:36.203135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:36.203110 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:03:36.203267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:36.203156 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 20:03:36.216189 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:36.216142 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podStartSLOduration=2.216128682 podStartE2EDuration="2.216128682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:36.214365637 +0000 UTC m=+914.731138224" watchObservedRunningTime="2026-11-02 20:03:36.216128682 +0000 UTC m=+914.732901267" Nov 02 20:03:37.202383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:37.202346 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:03:37.202806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:37.202399 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 20:03:38.201967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:38.201933 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:03:38.202250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:38.201999 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 20:03:39.201551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:39.201519 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:03:39.201959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:39.201575 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 20:03:39.301325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:39.301276 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6"] Nov 02 20:03:39.301557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:39.301529 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" containerID="cri-o://143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949" gracePeriod=30 Nov 02 20:03:40.201958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:40.201919 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:03:40.202400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:40.201984 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 20:03:41.055307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.054741 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss"] Nov 02 20:03:41.059884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.059809 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.100440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.100394 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jnr\" (UniqueName: \"kubernetes.io/projected/884c1be9-31f9-47ff-b41a-930e5e39b15d-kube-api-access-l8jnr\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.100611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.100512 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.100611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.100551 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.102318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.102276 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss"] Nov 02 20:03:41.152643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.152604 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:03:41.152938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.152922 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:03:41.153153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.153135 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:03:41.156094 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.156077 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:03:41.201429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.201396 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:03:41.201604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.201447 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 20:03:41.201604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.201517 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.201604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.201566 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.201727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.201682 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jnr\" (UniqueName: \"kubernetes.io/projected/884c1be9-31f9-47ff-b41a-930e5e39b15d-kube-api-access-l8jnr\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.202391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.202371 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.204254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.204229 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.256833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.256798 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jnr\" (UniqueName: \"kubernetes.io/projected/884c1be9-31f9-47ff-b41a-930e5e39b15d-kube-api-access-l8jnr\") pod \"maas-api-revoke-sub-6b149fe1-x89ss\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.374985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.374887 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:41.763122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:41.763087 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss"] Nov 02 20:03:42.179698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:42.179640 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" event={"ID":"884c1be9-31f9-47ff-b41a-930e5e39b15d","Type":"ContainerStarted","Data":"5a620c64425eb17dfe86b1e259f38e833fae82f30d6c4e6be553e04c1412fcf1"} Nov 02 20:03:42.179917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:42.179725 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" event={"ID":"884c1be9-31f9-47ff-b41a-930e5e39b15d","Type":"ContainerStarted","Data":"27dd4af2a007875c7c38ea1bdc510ee79dd8e5eff439a1545f0c7f0efaa0babc"} Nov 02 20:03:42.296790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:42.296666 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" podStartSLOduration=1.296644239 podStartE2EDuration="1.296644239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:42.292099041 +0000 UTC m=+920.808871631" watchObservedRunningTime="2026-11-02 20:03:42.296644239 +0000 UTC m=+920.813416828" Nov 02 20:03:43.199186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.199094 5800 generic.go:356] "Generic (PLEG): container finished" podID="884c1be9-31f9-47ff-b41a-930e5e39b15d" containerID="5a620c64425eb17dfe86b1e259f38e833fae82f30d6c4e6be553e04c1412fcf1" exitCode=0 Nov 02 20:03:43.199186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.199164 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" event={"ID":"884c1be9-31f9-47ff-b41a-930e5e39b15d","Type":"ContainerDied","Data":"5a620c64425eb17dfe86b1e259f38e833fae82f30d6c4e6be553e04c1412fcf1"} Nov 02 20:03:43.246079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.245413 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:03:43.246079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.245513 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:03:43.398401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.398360 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:43.398853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.398768 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:43.399540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.399513 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:03:43.610875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.610842 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-slhh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:03:43.611045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:43.610896 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:03:44.435182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.435153 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:44.580326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.578120 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 20:03:44.589511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.589479 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-xks-ca\") pod \"884c1be9-31f9-47ff-b41a-930e5e39b15d\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " Nov 02 20:03:44.589725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.589551 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-openshift-ca\") pod \"884c1be9-31f9-47ff-b41a-930e5e39b15d\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " Nov 02 20:03:44.589725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.589600 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884c1be9-31f9-47ff-b41a-930e5e39b15d-kube-api-access-l8jnr\" (UniqueName: \"kubernetes.io/projected/884c1be9-31f9-47ff-b41a-930e5e39b15d-kube-api-access-l8jnr\") pod \"884c1be9-31f9-47ff-b41a-930e5e39b15d\" (UID: \"884c1be9-31f9-47ff-b41a-930e5e39b15d\") " Nov 02 20:03:44.589911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.589884 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-openshift-ca" pod "884c1be9-31f9-47ff-b41a-930e5e39b15d" (UID: "884c1be9-31f9-47ff-b41a-930e5e39b15d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:44.590104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.590084 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.591688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.591659 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-xks-ca" pod "884c1be9-31f9-47ff-b41a-930e5e39b15d" (UID: "884c1be9-31f9-47ff-b41a-930e5e39b15d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:03:44.591824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.591800 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884c1be9-31f9-47ff-b41a-930e5e39b15d-kube-api-access-l8jnr" pod "884c1be9-31f9-47ff-b41a-930e5e39b15d" (UID: "884c1be9-31f9-47ff-b41a-930e5e39b15d"). InnerVolumeSpecName "kube-api-access-l8jnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:44.691200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691162 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-token\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-token\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.691442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691420 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-kube-api-access-qwq9l\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-kube-api-access-qwq9l\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.691678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691651 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istiod-ca-cert\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.691841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691720 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-data\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.691841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691750 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-socket\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.691841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691788 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-podinfo\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.691841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691835 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-credential-socket\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.692143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691908 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-certs\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.692143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.691939 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-envoy\") pod \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\" (UID: \"5dfdfd09-0256-4ca4-a4d1-c99ec864c822\") " Nov 02 20:03:44.692143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.692011 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istiod-ca-cert" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:03:44.692585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.692563 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-credential-socket" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.692810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.692661 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-data" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693030 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-certs" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693237 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884c1be9-31f9-47ff-b41a-930e5e39b15d-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693260 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693308 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8jnr\" (UniqueName: \"kubernetes.io/projected/884c1be9-31f9-47ff-b41a-930e5e39b15d-kube-api-access-l8jnr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693327 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693342 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693355 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.693452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.693423 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-socket" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.694129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.694090 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-token" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:44.694661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.694637 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-kube-api-access-qwq9l" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "kube-api-access-qwq9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:03:44.695720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.695694 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-envoy" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:03:44.696668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.696646 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-podinfo" pod "5dfdfd09-0256-4ca4-a4d1-c99ec864c822" (UID: "5dfdfd09-0256-4ca4-a4d1-c99ec864c822"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:03:44.794601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.794557 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.794601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.794589 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.794601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.794603 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.794926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.794615 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:44.794926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:44.794625 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwq9l\" (UniqueName: \"kubernetes.io/projected/5dfdfd09-0256-4ca4-a4d1-c99ec864c822-kube-api-access-qwq9l\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:03:45.211978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.211895 5800 generic.go:356] "Generic (PLEG): container finished" podID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerID="143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949" exitCode=0 Nov 02 20:03:45.211978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.211951 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" Nov 02 20:03:45.212198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.211992 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" event={"ID":"5dfdfd09-0256-4ca4-a4d1-c99ec864c822","Type":"ContainerDied","Data":"143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949"} Nov 02 20:03:45.212198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.212064 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6" event={"ID":"5dfdfd09-0256-4ca4-a4d1-c99ec864c822","Type":"ContainerDied","Data":"cfb0d29ef68e5b965f76cabccfadb57e3364daaecc7671228ef3c1a5cb74c96b"} Nov 02 20:03:45.212198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.212100 5800 scope.go:122] "RemoveContainer" containerID="143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949" Nov 02 20:03:45.213486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.213470 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" Nov 02 20:03:45.213574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.213500 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss" event={"ID":"884c1be9-31f9-47ff-b41a-930e5e39b15d","Type":"ContainerDied","Data":"27dd4af2a007875c7c38ea1bdc510ee79dd8e5eff439a1545f0c7f0efaa0babc"} Nov 02 20:03:45.213574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.213537 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27dd4af2a007875c7c38ea1bdc510ee79dd8e5eff439a1545f0c7f0efaa0babc" Nov 02 20:03:45.223542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.223517 5800 scope.go:122] "RemoveContainer" containerID="934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004" Nov 02 20:03:45.233053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.233031 5800 scope.go:122] "RemoveContainer" containerID="143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949" Nov 02 20:03:45.233378 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:03:45.233359 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949\": container with ID starting with 143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949 not found: ID does not exist" containerID="143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949" Nov 02 20:03:45.233444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.233384 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949"} err="failed to get container status \"143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949\": rpc error: code = NotFound desc = could not find container \"143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949\": container with ID starting with 143cbf57f50f8b1485dc77ec0f2df819281493ae7fd6c0bc76c9e02b784ec949 not found: ID does not exist" Nov 02 20:03:45.233444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.233403 5800 scope.go:122] "RemoveContainer" containerID="934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004" Nov 02 20:03:45.233637 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:03:45.233622 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004\": container with ID starting with 934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004 not found: ID does not exist" containerID="934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004" Nov 02 20:03:45.233674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.233641 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004"} err="failed to get container status \"934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004\": rpc error: code = NotFound desc = could not find container \"934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004\": container with ID starting with 934902db75d1e6f1221e7dc50b47bdd3df95cbe5be7fe2848ba4389897df9004 not found: ID does not exist" Nov 02 20:03:45.440055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.440021 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6"] Nov 02 20:03:45.456012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.455981 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-slhh6"] Nov 02 20:03:45.623993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:45.623956 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" path="/var/lib/kubelet/pods/5dfdfd09-0256-4ca4-a4d1-c99ec864c822/volumes" Nov 02 20:03:52.696354 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:03:52.696303 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:03:52.860860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.860819 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr"] Nov 02 20:03:52.861469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861439 5800 cpu_manager.go:414] "containerMap: removing container" podUID="884c1be9-31f9-47ff-b41a-930e5e39b15d" containerName="revoke-keys" Nov 02 20:03:52.861469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861460 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884c1be9-31f9-47ff-b41a-930e5e39b15d" containerName="revoke-keys" Nov 02 20:03:52.861639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861480 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" Nov 02 20:03:52.861639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861487 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" Nov 02 20:03:52.861639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861507 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" Nov 02 20:03:52.861639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861515 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" Nov 02 20:03:52.861835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861666 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="884c1be9-31f9-47ff-b41a-930e5e39b15d" containerName="revoke-keys" Nov 02 20:03:52.861835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861681 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" Nov 02 20:03:52.861835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.861692 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfdfd09-0256-4ca4-a4d1-c99ec864c822" containerName="istio-proxy" Nov 02 20:03:52.866526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.866503 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:52.919943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.919906 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr"] Nov 02 20:03:52.961849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.961758 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"premium-w1-bbee67-kserve-self-signed-certs\"" Nov 02 20:03:52.961849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.961830 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"default-dockercfg-gj7fz\"" Nov 02 20:03:52.961849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.961836 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"openshift-service-ca.crt\"" Nov 02 20:03:52.964052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.964036 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"kube-root-ca.crt\"" Nov 02 20:03:52.981044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.981017 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-tmp-dir\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:52.981166 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.981053 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6qm\" (UniqueName: \"kubernetes.io/projected/888e258a-9ac1-405c-8055-b00a43205cf8-kube-api-access-gn6qm\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:52.981166 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.981080 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888e258a-9ac1-405c-8055-b00a43205cf8-tls-certs\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:52.981272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.981249 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-model-cache\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:52.981378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.981365 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-home\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:52.981415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:52.981397 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-dshm\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.082725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.082694 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-dshm\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.082932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.082753 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-tmp-dir\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.082932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.082789 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6qm\" (UniqueName: \"kubernetes.io/projected/888e258a-9ac1-405c-8055-b00a43205cf8-kube-api-access-gn6qm\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.082932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.082821 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888e258a-9ac1-405c-8055-b00a43205cf8-tls-certs\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.083118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.082933 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-model-cache\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.083118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.083069 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-home\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.083235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.083213 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-tmp-dir\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.083429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.083403 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-home\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.083561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.083474 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-model-cache\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.085433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.085405 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-dshm\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.085575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.085554 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888e258a-9ac1-405c-8055-b00a43205cf8-tls-certs\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.135939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.135908 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6qm\" (UniqueName: \"kubernetes.io/projected/888e258a-9ac1-405c-8055-b00a43205cf8-kube-api-access-gn6qm\") pod \"premium-w1-bbee67-kserve-55d59456c4-dz8zr\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.181679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.181643 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:03:53.580505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:53.580481 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr"] Nov 02 20:03:53.582009 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:03:53.581976 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888e258a_9ac1_405c_8055_b00a43205cf8.slice/crio-5a9da710f8529532f5bd57c228172583fff0363e3549ad56477592edeefb78ff WatchSource:0}: Error finding container 5a9da710f8529532f5bd57c228172583fff0363e3549ad56477592edeefb78ff: Status 404 returned error can't find the container with id 5a9da710f8529532f5bd57c228172583fff0363e3549ad56477592edeefb78ff Nov 02 20:03:54.297762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:54.297723 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" event={"ID":"888e258a-9ac1-405c-8055-b00a43205cf8","Type":"ContainerStarted","Data":"6ea2e58446cab6105a5639a1bd6a3f91ba16334297009aa57a971670108b4f5e"} Nov 02 20:03:54.298271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:54.297771 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" event={"ID":"888e258a-9ac1-405c-8055-b00a43205cf8","Type":"ContainerStarted","Data":"5a9da710f8529532f5bd57c228172583fff0363e3549ad56477592edeefb78ff"} Nov 02 20:03:54.404563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:54.404508 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podStartSLOduration=2.404494374 podStartE2EDuration="2.404494374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:03:54.403149497 +0000 UTC m=+932.919922082" watchObservedRunningTime="2026-11-02 20:03:54.404494374 +0000 UTC m=+932.921266964" Nov 02 20:03:55.189752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:03:55.189717 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:04:03.181811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:03.181777 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:04:03.196960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:03.196930 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:04:03.375662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:03.375631 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:04:04.930138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:04.930104 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb"] Nov 02 20:04:04.934133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:04.934109 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:04.984024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:04.983983 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:04:04.984264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:04.984075 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:04:05.030021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.029982 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:04:05.030229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.030052 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:04:05.032763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.032735 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb"] Nov 02 20:04:05.032912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.032891 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.033011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.032997 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.033093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.033080 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlg8l\" (UniqueName: \"kubernetes.io/projected/84528565-5aba-4954-af8d-4c77a3307c8d-kube-api-access-zlg8l\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.133956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.133920 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlg8l\" (UniqueName: \"kubernetes.io/projected/84528565-5aba-4954-af8d-4c77a3307c8d-kube-api-access-zlg8l\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.134130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.133996 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.134130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.134056 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.134771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.134751 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.136342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.136313 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.187210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.187132 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlg8l\" (UniqueName: \"kubernetes.io/projected/84528565-5aba-4954-af8d-4c77a3307c8d-kube-api-access-zlg8l\") pod \"maas-api-revoke-sub-125df16a-pg7mb\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.248199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.248167 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:05.616035 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:04:05.615997 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84528565_5aba_4954_af8d_4c77a3307c8d.slice/crio-8c9498e7baf17183058d444b0665ed1954ca0181f2b1757e7e230f3e3cec525a WatchSource:0}: Error finding container 8c9498e7baf17183058d444b0665ed1954ca0181f2b1757e7e230f3e3cec525a: Status 404 returned error can't find the container with id 8c9498e7baf17183058d444b0665ed1954ca0181f2b1757e7e230f3e3cec525a Nov 02 20:04:05.623073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:05.623042 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb"] Nov 02 20:04:06.384653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:06.384612 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" event={"ID":"84528565-5aba-4954-af8d-4c77a3307c8d","Type":"ContainerStarted","Data":"2d69e4d4c359f2c77bea5d77262ca8103897bce740357076b410d90fc0079284"} Nov 02 20:04:06.385144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:06.384674 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" event={"ID":"84528565-5aba-4954-af8d-4c77a3307c8d","Type":"ContainerStarted","Data":"8c9498e7baf17183058d444b0665ed1954ca0181f2b1757e7e230f3e3cec525a"} Nov 02 20:04:06.494494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:06.494437 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" podStartSLOduration=2.494417796 podStartE2EDuration="2.494417796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:06.489256804 +0000 UTC m=+945.006029390" watchObservedRunningTime="2026-11-02 20:04:06.494417796 +0000 UTC m=+945.011190384" Nov 02 20:04:07.392432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:07.392397 5800 generic.go:356] "Generic (PLEG): container finished" podID="84528565-5aba-4954-af8d-4c77a3307c8d" containerID="2d69e4d4c359f2c77bea5d77262ca8103897bce740357076b410d90fc0079284" exitCode=0 Nov 02 20:04:07.392824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:07.392484 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" event={"ID":"84528565-5aba-4954-af8d-4c77a3307c8d","Type":"ContainerDied","Data":"2d69e4d4c359f2c77bea5d77262ca8103897bce740357076b410d90fc0079284"} Nov 02 20:04:08.551392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.551368 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:08.671699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.671608 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-openshift-ca\") pod \"84528565-5aba-4954-af8d-4c77a3307c8d\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " Nov 02 20:04:08.671699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.671662 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-xks-ca\") pod \"84528565-5aba-4954-af8d-4c77a3307c8d\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " Nov 02 20:04:08.671938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.671886 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84528565-5aba-4954-af8d-4c77a3307c8d-kube-api-access-zlg8l\" (UniqueName: \"kubernetes.io/projected/84528565-5aba-4954-af8d-4c77a3307c8d-kube-api-access-zlg8l\") pod \"84528565-5aba-4954-af8d-4c77a3307c8d\" (UID: \"84528565-5aba-4954-af8d-4c77a3307c8d\") " Nov 02 20:04:08.672075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.672043 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-openshift-ca" pod "84528565-5aba-4954-af8d-4c77a3307c8d" (UID: "84528565-5aba-4954-af8d-4c77a3307c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:08.672309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.672273 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:08.673892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.673869 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-xks-ca" pod "84528565-5aba-4954-af8d-4c77a3307c8d" (UID: "84528565-5aba-4954-af8d-4c77a3307c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:08.676405 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.676384 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84528565-5aba-4954-af8d-4c77a3307c8d-kube-api-access-zlg8l" pod "84528565-5aba-4954-af8d-4c77a3307c8d" (UID: "84528565-5aba-4954-af8d-4c77a3307c8d"). InnerVolumeSpecName "kube-api-access-zlg8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:08.773904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.773866 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlg8l\" (UniqueName: \"kubernetes.io/projected/84528565-5aba-4954-af8d-4c77a3307c8d-kube-api-access-zlg8l\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:08.773904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:08.773899 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84528565-5aba-4954-af8d-4c77a3307c8d-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:09.408236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:09.408205 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" Nov 02 20:04:09.408443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:09.408242 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb" event={"ID":"84528565-5aba-4954-af8d-4c77a3307c8d","Type":"ContainerDied","Data":"8c9498e7baf17183058d444b0665ed1954ca0181f2b1757e7e230f3e3cec525a"} Nov 02 20:04:09.408443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:09.408307 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c9498e7baf17183058d444b0665ed1954ca0181f2b1757e7e230f3e3cec525a" Nov 02 20:04:20.419703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.419665 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj"] Nov 02 20:04:20.420206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.420146 5800 cpu_manager.go:414] "containerMap: removing container" podUID="84528565-5aba-4954-af8d-4c77a3307c8d" containerName="revoke-keys" Nov 02 20:04:20.420206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.420158 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84528565-5aba-4954-af8d-4c77a3307c8d" containerName="revoke-keys" Nov 02 20:04:20.420318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.420313 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="84528565-5aba-4954-af8d-4c77a3307c8d" containerName="revoke-keys" Nov 02 20:04:20.424723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.424705 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.475443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.475410 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:04:20.498364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.498333 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.498364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.498367 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.498636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.498395 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqtm\" (UniqueName: \"kubernetes.io/projected/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-kube-api-access-tsqtm\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.521463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.521433 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:04:20.521641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.521568 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:04:20.521641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.521582 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:04:20.525749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.525727 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj"] Nov 02 20:04:20.599848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.599816 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.599848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.599848 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.600114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.599872 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqtm\" (UniqueName: \"kubernetes.io/projected/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-kube-api-access-tsqtm\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.600601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.600573 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.602303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.602269 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.653196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.653165 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqtm\" (UniqueName: \"kubernetes.io/projected/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-kube-api-access-tsqtm\") pod \"maas-api-revoke-sub-dcb0715a-jjzjj\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:20.737665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:20.737577 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:21.109319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:21.109261 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj"] Nov 02 20:04:21.110583 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:04:21.110555 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf290f2f3_bef2_41d7_a1ea_65b9584bfc09.slice/crio-9a8382c57197968550e89420c5f1fc97ec4857f035579c72c7f0fdb8d1d14c1b WatchSource:0}: Error finding container 9a8382c57197968550e89420c5f1fc97ec4857f035579c72c7f0fdb8d1d14c1b: Status 404 returned error can't find the container with id 9a8382c57197968550e89420c5f1fc97ec4857f035579c72c7f0fdb8d1d14c1b Nov 02 20:04:21.488066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:21.488024 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" event={"ID":"f290f2f3-bef2-41d7-a1ea-65b9584bfc09","Type":"ContainerStarted","Data":"84ca21d14c8a8ae00c2dd12f206466c1602a08e9e71960f1c52604adecdc70ad"} Nov 02 20:04:21.488480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:21.488083 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" event={"ID":"f290f2f3-bef2-41d7-a1ea-65b9584bfc09","Type":"ContainerStarted","Data":"9a8382c57197968550e89420c5f1fc97ec4857f035579c72c7f0fdb8d1d14c1b"} Nov 02 20:04:21.600788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:21.600735 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" podStartSLOduration=1.600721643 podStartE2EDuration="1.600721643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:21.598975168 +0000 UTC m=+960.115747752" watchObservedRunningTime="2026-11-02 20:04:21.600721643 +0000 UTC m=+960.117494228" Nov 02 20:04:22.495635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:22.495597 5800 generic.go:356] "Generic (PLEG): container finished" podID="f290f2f3-bef2-41d7-a1ea-65b9584bfc09" containerID="84ca21d14c8a8ae00c2dd12f206466c1602a08e9e71960f1c52604adecdc70ad" exitCode=0 Nov 02 20:04:22.496046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:22.495684 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" event={"ID":"f290f2f3-bef2-41d7-a1ea-65b9584bfc09","Type":"ContainerDied","Data":"84ca21d14c8a8ae00c2dd12f206466c1602a08e9e71960f1c52604adecdc70ad"} Nov 02 20:04:22.697657 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:04:22.697624 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:04:23.764547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.764452 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:23.861692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.861652 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-openshift-ca\") pod \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " Nov 02 20:04:23.861899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.861751 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-xks-ca\") pod \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " Nov 02 20:04:23.861899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.861789 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-kube-api-access-tsqtm\" (UniqueName: \"kubernetes.io/projected/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-kube-api-access-tsqtm\") pod \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\" (UID: \"f290f2f3-bef2-41d7-a1ea-65b9584bfc09\") " Nov 02 20:04:23.863155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.863123 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-openshift-ca" pod "f290f2f3-bef2-41d7-a1ea-65b9584bfc09" (UID: "f290f2f3-bef2-41d7-a1ea-65b9584bfc09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:23.865462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.865437 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-kube-api-access-tsqtm" pod "f290f2f3-bef2-41d7-a1ea-65b9584bfc09" (UID: "f290f2f3-bef2-41d7-a1ea-65b9584bfc09"). InnerVolumeSpecName "kube-api-access-tsqtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:23.877418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.877390 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-xks-ca" pod "f290f2f3-bef2-41d7-a1ea-65b9584bfc09" (UID: "f290f2f3-bef2-41d7-a1ea-65b9584bfc09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:23.964305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.964219 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:23.964305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.964269 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:23.964305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:23.964302 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqtm\" (UniqueName: \"kubernetes.io/projected/f290f2f3-bef2-41d7-a1ea-65b9584bfc09-kube-api-access-tsqtm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:24.520513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:24.520473 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" event={"ID":"f290f2f3-bef2-41d7-a1ea-65b9584bfc09","Type":"ContainerDied","Data":"9a8382c57197968550e89420c5f1fc97ec4857f035579c72c7f0fdb8d1d14c1b"} Nov 02 20:04:24.520513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:24.520519 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj" Nov 02 20:04:24.520752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:24.520535 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a8382c57197968550e89420c5f1fc97ec4857f035579c72c7f0fdb8d1d14c1b" Nov 02 20:04:25.552416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.552382 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l"] Nov 02 20:04:25.552893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.552880 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f290f2f3-bef2-41d7-a1ea-65b9584bfc09" containerName="revoke-keys" Nov 02 20:04:25.552937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.552894 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f290f2f3-bef2-41d7-a1ea-65b9584bfc09" containerName="revoke-keys" Nov 02 20:04:25.552992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.552983 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f290f2f3-bef2-41d7-a1ea-65b9584bfc09" containerName="revoke-keys" Nov 02 20:04:25.556501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.556479 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.558254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.558233 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4"] Nov 02 20:04:25.561943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.561918 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.605848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.605814 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 20:04:25.606076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.605814 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 20:04:25.606076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.605825 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 20:04:25.606076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.605839 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 20:04:25.652073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.652040 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e2170d8c02f2\"" Nov 02 20:04:25.652073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.652040 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-543f396ff8c5\"" Nov 02 20:04:25.652318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.652046 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5p6z6\"" Nov 02 20:04:25.655762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.655732 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4"] Nov 02 20:04:25.658343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.658317 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l"] Nov 02 20:04:25.692707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.692665 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgmfd\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-kube-api-access-vgmfd\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.692895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.692772 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-routing-overlay-volume\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.692895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.692849 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.693015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.692905 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxsqh\" (UniqueName: \"kubernetes.io/projected/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-kube-api-access-pxsqh\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.693015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.692931 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.693119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.693030 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.693119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.693063 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-plugins-config-volume\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.705894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.705864 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:04:25.706081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.705925 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:04:25.706081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.705933 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:04:25.706081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.706014 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:04:25.794403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.794370 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.794667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.794414 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-plugins-config-volume\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.794667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.794488 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgmfd\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-kube-api-access-vgmfd\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.794667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.794529 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-routing-overlay-volume\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.794667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.794549 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.794667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.794578 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxsqh\" (UniqueName: \"kubernetes.io/projected/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-kube-api-access-pxsqh\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.794667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.794601 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.795125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.795091 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.795260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.795218 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-routing-overlay-volume\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.795364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.795349 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-plugins-config-volume\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.797556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.797531 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.797859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.797840 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.848709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.848632 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgmfd\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-kube-api-access-vgmfd\") pod \"payload-processing-external-model-74f4f7968-rjh4l\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.848856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.848770 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxsqh\" (UniqueName: \"kubernetes.io/projected/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-kube-api-access-pxsqh\") pod \"maas-api-revoke-sub-b7a49c91-kxgc4\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:25.873740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.873707 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:25.884687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:25.884618 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:26.292476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:26.292450 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l"] Nov 02 20:04:26.294178 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:04:26.294146 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf500490_3634_4b1f_a950_7cc11ef7fe12.slice/crio-757476f82ad7db328f25ec5b087fd4d6f97b3e3acfc0a165382b9a21be270b3d WatchSource:0}: Error finding container 757476f82ad7db328f25ec5b087fd4d6f97b3e3acfc0a165382b9a21be270b3d: Status 404 returned error can't find the container with id 757476f82ad7db328f25ec5b087fd4d6f97b3e3acfc0a165382b9a21be270b3d Nov 02 20:04:26.315054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:26.315030 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4"] Nov 02 20:04:26.333117 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:04:26.333088 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd6d7ff_66ac_44a9_8dad_12f25d67bfd0.slice/crio-6b3cbad6826b7a61ca1d0513116127edb7ec35d0f0cfea964cb1cf923fa760a7 WatchSource:0}: Error finding container 6b3cbad6826b7a61ca1d0513116127edb7ec35d0f0cfea964cb1cf923fa760a7: Status 404 returned error can't find the container with id 6b3cbad6826b7a61ca1d0513116127edb7ec35d0f0cfea964cb1cf923fa760a7 Nov 02 20:04:26.550113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:26.550015 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" event={"ID":"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0","Type":"ContainerStarted","Data":"6b3cbad6826b7a61ca1d0513116127edb7ec35d0f0cfea964cb1cf923fa760a7"} Nov 02 20:04:26.552238 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:26.552206 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" event={"ID":"bf500490-3634-4b1f-a950-7cc11ef7fe12","Type":"ContainerStarted","Data":"10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae"} Nov 02 20:04:26.552398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:26.552258 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" event={"ID":"bf500490-3634-4b1f-a950-7cc11ef7fe12","Type":"ContainerStarted","Data":"757476f82ad7db328f25ec5b087fd4d6f97b3e3acfc0a165382b9a21be270b3d"} Nov 02 20:04:26.552574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:26.552554 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:26.660636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:26.660583 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" podStartSLOduration=1.6605688870000002 podStartE2EDuration="1.660568887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:26.658489216 +0000 UTC m=+965.175261801" watchObservedRunningTime="2026-11-02 20:04:26.660568887 +0000 UTC m=+965.177341472" Nov 02 20:04:27.566553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:27.566457 5800 generic.go:356] "Generic (PLEG): container finished" podID="fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" containerID="cee557e123bba05ef03eeb828f610950af5068a1b600bd6ad6214454a922b0c8" exitCode=0 Nov 02 20:04:27.567427 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:27.567399 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" event={"ID":"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0","Type":"ContainerDied","Data":"cee557e123bba05ef03eeb828f610950af5068a1b600bd6ad6214454a922b0c8"} Nov 02 20:04:28.715109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.715083 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:28.833729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.833636 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-openshift-ca\") pod \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " Nov 02 20:04:28.833914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.833771 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-xks-ca\") pod \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " Nov 02 20:04:28.833914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.833883 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-kube-api-access-pxsqh\" (UniqueName: \"kubernetes.io/projected/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-kube-api-access-pxsqh\") pod \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\" (UID: \"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0\") " Nov 02 20:04:28.834119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.834041 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-openshift-ca" pod "fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" (UID: "fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:28.834374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.834352 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:28.836487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.836461 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-xks-ca" pod "fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" (UID: "fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:28.836487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.836477 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-kube-api-access-pxsqh" pod "fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" (UID: "fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0"). InnerVolumeSpecName "kube-api-access-pxsqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:28.935327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.935275 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:28.935327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:28.935318 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxsqh\" (UniqueName: \"kubernetes.io/projected/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0-kube-api-access-pxsqh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:29.583259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:29.583229 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" Nov 02 20:04:29.583460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:29.583252 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4" event={"ID":"fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0","Type":"ContainerDied","Data":"6b3cbad6826b7a61ca1d0513116127edb7ec35d0f0cfea964cb1cf923fa760a7"} Nov 02 20:04:29.583460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:29.583317 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3cbad6826b7a61ca1d0513116127edb7ec35d0f0cfea964cb1cf923fa760a7" Nov 02 20:04:36.376257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.376219 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh"] Nov 02 20:04:36.376764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.376748 5800 cpu_manager.go:414] "containerMap: removing container" podUID="fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" containerName="revoke-keys" Nov 02 20:04:36.376809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.376765 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" containerName="revoke-keys" Nov 02 20:04:36.376887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.376877 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" containerName="revoke-keys" Nov 02 20:04:36.380133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.380118 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.414855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.414824 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.415022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.414883 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpsdl\" (UniqueName: \"kubernetes.io/projected/76e8d8c5-08da-4acf-bd63-a561b23292b2-kube-api-access-mpsdl\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.415022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.414987 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.433641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.433604 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh"] Nov 02 20:04:36.475602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.475560 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:04:36.475602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.475572 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:04:36.475602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.475573 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:04:36.476568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.476544 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:04:36.516327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.516265 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.516512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.516379 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpsdl\" (UniqueName: \"kubernetes.io/projected/76e8d8c5-08da-4acf-bd63-a561b23292b2-kube-api-access-mpsdl\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.516512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.516433 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.517086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.517063 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.518536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.518512 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.580001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.579971 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpsdl\" (UniqueName: \"kubernetes.io/projected/76e8d8c5-08da-4acf-bd63-a561b23292b2-kube-api-access-mpsdl\") pod \"maas-api-revoke-sub-13f53506-n6xsh\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:36.693454 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:36.693373 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:37.062876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:37.062843 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh"] Nov 02 20:04:37.063621 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:04:37.063588 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e8d8c5_08da_4acf_bd63_a561b23292b2.slice/crio-62d0fa300275282359e0df4d88808a53db9e3a39a62aee8044ba392603dcefcf WatchSource:0}: Error finding container 62d0fa300275282359e0df4d88808a53db9e3a39a62aee8044ba392603dcefcf: Status 404 returned error can't find the container with id 62d0fa300275282359e0df4d88808a53db9e3a39a62aee8044ba392603dcefcf Nov 02 20:04:37.570638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:37.570612 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:04:37.635762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:37.635683 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" event={"ID":"76e8d8c5-08da-4acf-bd63-a561b23292b2","Type":"ContainerStarted","Data":"5dc8bdb24f6ff6f8ea6212d0caeb5a75a092d8baddeab0b8ab93068f9a5c46b3"} Nov 02 20:04:37.635762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:37.635732 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" event={"ID":"76e8d8c5-08da-4acf-bd63-a561b23292b2","Type":"ContainerStarted","Data":"62d0fa300275282359e0df4d88808a53db9e3a39a62aee8044ba392603dcefcf"} Nov 02 20:04:37.786204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:37.786139 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" podStartSLOduration=1.7861203049999999 podStartE2EDuration="1.786120305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:37.782800593 +0000 UTC m=+976.299573179" watchObservedRunningTime="2026-11-02 20:04:37.786120305 +0000 UTC m=+976.302892891" Nov 02 20:04:38.647528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:38.647481 5800 generic.go:356] "Generic (PLEG): container finished" podID="76e8d8c5-08da-4acf-bd63-a561b23292b2" containerID="5dc8bdb24f6ff6f8ea6212d0caeb5a75a092d8baddeab0b8ab93068f9a5c46b3" exitCode=0 Nov 02 20:04:38.648048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:38.647653 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" event={"ID":"76e8d8c5-08da-4acf-bd63-a561b23292b2","Type":"ContainerDied","Data":"5dc8bdb24f6ff6f8ea6212d0caeb5a75a092d8baddeab0b8ab93068f9a5c46b3"} Nov 02 20:04:39.854471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.854440 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:39.959151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.959047 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-openshift-ca\") pod \"76e8d8c5-08da-4acf-bd63-a561b23292b2\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " Nov 02 20:04:39.959151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.959107 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e8d8c5-08da-4acf-bd63-a561b23292b2-kube-api-access-mpsdl\" (UniqueName: \"kubernetes.io/projected/76e8d8c5-08da-4acf-bd63-a561b23292b2-kube-api-access-mpsdl\") pod \"76e8d8c5-08da-4acf-bd63-a561b23292b2\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " Nov 02 20:04:39.959498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.959254 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-xks-ca\") pod \"76e8d8c5-08da-4acf-bd63-a561b23292b2\" (UID: \"76e8d8c5-08da-4acf-bd63-a561b23292b2\") " Nov 02 20:04:39.959624 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.959520 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-openshift-ca" pod "76e8d8c5-08da-4acf-bd63-a561b23292b2" (UID: "76e8d8c5-08da-4acf-bd63-a561b23292b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:39.959936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.959766 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:39.962558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.962518 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-xks-ca" pod "76e8d8c5-08da-4acf-bd63-a561b23292b2" (UID: "76e8d8c5-08da-4acf-bd63-a561b23292b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:39.962980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:39.962955 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e8d8c5-08da-4acf-bd63-a561b23292b2-kube-api-access-mpsdl" pod "76e8d8c5-08da-4acf-bd63-a561b23292b2" (UID: "76e8d8c5-08da-4acf-bd63-a561b23292b2"). InnerVolumeSpecName "kube-api-access-mpsdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:40.061133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:40.061077 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e8d8c5-08da-4acf-bd63-a561b23292b2-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:40.061133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:40.061125 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpsdl\" (UniqueName: \"kubernetes.io/projected/76e8d8c5-08da-4acf-bd63-a561b23292b2-kube-api-access-mpsdl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:40.665350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:40.665320 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" Nov 02 20:04:40.665350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:40.665342 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh" event={"ID":"76e8d8c5-08da-4acf-bd63-a561b23292b2","Type":"ContainerDied","Data":"62d0fa300275282359e0df4d88808a53db9e3a39a62aee8044ba392603dcefcf"} Nov 02 20:04:40.665610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:40.665385 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62d0fa300275282359e0df4d88808a53db9e3a39a62aee8044ba392603dcefcf" Nov 02 20:04:45.118823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.118771 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7"] Nov 02 20:04:45.119583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.119565 5800 cpu_manager.go:414] "containerMap: removing container" podUID="76e8d8c5-08da-4acf-bd63-a561b23292b2" containerName="revoke-keys" Nov 02 20:04:45.119637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.119586 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e8d8c5-08da-4acf-bd63-a561b23292b2" containerName="revoke-keys" Nov 02 20:04:45.119747 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.119736 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e8d8c5-08da-4acf-bd63-a561b23292b2" containerName="revoke-keys" Nov 02 20:04:45.123501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.123478 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.174512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.174485 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7"] Nov 02 20:04:45.222859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.222817 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-routing-overlay-volume\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.223028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.222878 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-provider-credentials\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.223028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.222991 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-plugins-config-volume\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.223166 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.223144 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6smb\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-kube-api-access-c6smb\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.324136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.324100 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6smb\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-kube-api-access-c6smb\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.324347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.324154 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-routing-overlay-volume\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.324347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.324181 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-provider-credentials\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.324347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.324244 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-plugins-config-volume\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.324880 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.324854 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-routing-overlay-volume\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.325001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.324979 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-plugins-config-volume\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.326694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.326672 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-provider-credentials\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.377150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.377077 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6smb\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-kube-api-access-c6smb\") pod \"payload-processing-external-model-6ccc49f6cb-mnpl7\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.437630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.437590 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:45.806971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:45.806941 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7"] Nov 02 20:04:45.809296 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:04:45.809257 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4b868a_a9a6_429a_be54_14a73a45f7a1.slice/crio-c4d7b4261321b8716e1bbd28489b033d577a015b634ff89e8ea2005b6b412fad WatchSource:0}: Error finding container c4d7b4261321b8716e1bbd28489b033d577a015b634ff89e8ea2005b6b412fad: Status 404 returned error can't find the container with id c4d7b4261321b8716e1bbd28489b033d577a015b634ff89e8ea2005b6b412fad Nov 02 20:04:46.715092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:46.715053 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" event={"ID":"6b4b868a-a9a6-429a-be54-14a73a45f7a1","Type":"ContainerStarted","Data":"aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb"} Nov 02 20:04:46.715573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:46.715110 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" event={"ID":"6b4b868a-a9a6-429a-be54-14a73a45f7a1","Type":"ContainerStarted","Data":"c4d7b4261321b8716e1bbd28489b033d577a015b634ff89e8ea2005b6b412fad"} Nov 02 20:04:46.715573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:46.715147 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:04:46.821760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:46.821710 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" podStartSLOduration=1.821695471 podStartE2EDuration="1.821695471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:46.820412859 +0000 UTC m=+985.337185445" watchObservedRunningTime="2026-11-02 20:04:46.821695471 +0000 UTC m=+985.338468046" Nov 02 20:04:52.699235 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:04:52.699196 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:04:54.123176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.123139 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52"] Nov 02 20:04:54.127527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.127506 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.179964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.179935 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52"] Nov 02 20:04:54.222803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.222773 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:04:54.222803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.222791 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:04:54.222803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.222793 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:04:54.223221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.223203 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:04:54.227691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.227666 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.227836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.227819 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9fh\" (UniqueName: \"kubernetes.io/projected/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-kube-api-access-gv9fh\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.227960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.227943 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.329044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.329006 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.329271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.329061 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.329271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.329143 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9fh\" (UniqueName: \"kubernetes.io/projected/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-kube-api-access-gv9fh\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.329918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.329896 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.331526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.331502 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.382243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.382164 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9fh\" (UniqueName: \"kubernetes.io/projected/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-kube-api-access-gv9fh\") pod \"maas-api-revoke-sub-fd558761-qfb52\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.441758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.441724 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:54.811570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.811540 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52"] Nov 02 20:04:54.886192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.886152 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw"] Nov 02 20:04:54.893604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.893576 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:54.918807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.918779 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q"] Nov 02 20:04:54.922860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.922842 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:54.977367 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:54.977336 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw"] Nov 02 20:04:55.024074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.024042 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q"] Nov 02 20:04:55.036585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036550 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b06b7-be99-435c-a148-0bbe1620cb50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.036752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036603 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.036752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036629 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.036853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036775 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.036853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036803 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.036853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036822 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.036955 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036906 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.036987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.036966 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.037023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037010 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.037054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037032 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.037129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037111 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.037165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037138 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5p2k\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-kube-api-access-l5p2k\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.037198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037175 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.037243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037231 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.037297 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037254 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.037335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037321 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.037366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037339 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b06b7-be99-435c-a148-0bbe1620cb50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.037366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.037360 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8nt\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-kube-api-access-kq8nt\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.137999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.137964 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138013 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138141 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138201 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138240 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b06b7-be99-435c-a148-0bbe1620cb50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138274 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8nt\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-kube-api-access-kq8nt\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138340 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b06b7-be99-435c-a148-0bbe1620cb50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138380 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138406 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138472 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138498 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138522 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138547 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138548 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138640 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138688 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138716 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138781 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138829 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5p2k\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-kube-api-access-l5p2k\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138859 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.138921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.138877 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.139615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.139014 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.139615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.139141 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.139615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.139187 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.139615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.139187 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.139615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.139276 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.139935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.139803 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.140241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.140212 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b06b7-be99-435c-a148-0bbe1620cb50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.140603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.140580 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.142025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.142006 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b06b7-be99-435c-a148-0bbe1620cb50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.142233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.142207 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.143267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.143247 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.191241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.191200 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.192334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.192309 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5p2k\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-kube-api-access-l5p2k\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.192445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.192393 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8nt\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-kube-api-access-kq8nt\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9vr7q\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.192445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.192412 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-jb7jw\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.207230 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.207204 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:55.235817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.235787 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:55.650875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.650847 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw"] Nov 02 20:04:55.650992 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:04:55.650957 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b65472e_f834_406f_ae97_d4aecaa5e1e4.slice/crio-e3d8fa656b9c3d9188812879e3867bd3f5dc8f2ef3dcb2f500dbad687c4ab4c0 WatchSource:0}: Error finding container e3d8fa656b9c3d9188812879e3867bd3f5dc8f2ef3dcb2f500dbad687c4ab4c0: Status 404 returned error can't find the container with id e3d8fa656b9c3d9188812879e3867bd3f5dc8f2ef3dcb2f500dbad687c4ab4c0 Nov 02 20:04:55.651210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.651179 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:55.651343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.651242 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:55.651343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.651270 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:55.653446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.653421 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:55.653541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.653490 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:55.653622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.653546 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:04:55.653848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.653773 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q"] Nov 02 20:04:55.783319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.783256 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" event={"ID":"6b65472e-f834-406f-ae97-d4aecaa5e1e4","Type":"ContainerStarted","Data":"1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08"} Nov 02 20:04:55.784738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.784712 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" event={"ID":"6b65472e-f834-406f-ae97-d4aecaa5e1e4","Type":"ContainerStarted","Data":"e3d8fa656b9c3d9188812879e3867bd3f5dc8f2ef3dcb2f500dbad687c4ab4c0"} Nov 02 20:04:55.786416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.786389 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" event={"ID":"284b06b7-be99-435c-a148-0bbe1620cb50","Type":"ContainerStarted","Data":"b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c"} Nov 02 20:04:55.786541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.786441 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" event={"ID":"284b06b7-be99-435c-a148-0bbe1620cb50","Type":"ContainerStarted","Data":"633d323529c16fac0835730d2ba2a1078944966567f0261cae6d3f228bccdc72"} Nov 02 20:04:55.789198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.789175 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" event={"ID":"f95f78e2-e9d7-450f-afb4-144cdaa5ab28","Type":"ContainerStarted","Data":"0a2dfc3cf4ed9d5dcfbb47fcb5a3a7851f3a2977e59e6acb45d3c9d27fa3f7ab"} Nov 02 20:04:55.789301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.789221 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" event={"ID":"f95f78e2-e9d7-450f-afb4-144cdaa5ab28","Type":"ContainerStarted","Data":"0fdafa8728c20d46a685ddfd8c087c8816776c0379cf5a3d36d2b8568ea72135"} Nov 02 20:04:55.892955 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:55.892907 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podStartSLOduration=1.892890349 podStartE2EDuration="1.892890349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:55.892739059 +0000 UTC m=+994.409511644" watchObservedRunningTime="2026-11-02 20:04:55.892890349 +0000 UTC m=+994.409662932" Nov 02 20:04:56.006613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.006555 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podStartSLOduration=2.006535044 podStartE2EDuration="2.006535044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:56.002933849 +0000 UTC m=+994.519706448" watchObservedRunningTime="2026-11-02 20:04:56.006535044 +0000 UTC m=+994.523307632" Nov 02 20:04:56.114513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.114460 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" podStartSLOduration=2.114445228 podStartE2EDuration="2.114445228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:04:56.114176016 +0000 UTC m=+994.630948600" watchObservedRunningTime="2026-11-02 20:04:56.114445228 +0000 UTC m=+994.631217812" Nov 02 20:04:56.210752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.208685 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:04:56.214448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.214410 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:04:56.214707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.214462 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:04:56.246317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.236782 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:04:56.250312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.248414 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:04:56.250312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.248475 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:04:56.801307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.801248 5800 generic.go:356] "Generic (PLEG): container finished" podID="f95f78e2-e9d7-450f-afb4-144cdaa5ab28" containerID="0a2dfc3cf4ed9d5dcfbb47fcb5a3a7851f3a2977e59e6acb45d3c9d27fa3f7ab" exitCode=0 Nov 02 20:04:56.801915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:56.801752 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" event={"ID":"f95f78e2-e9d7-450f-afb4-144cdaa5ab28","Type":"ContainerDied","Data":"0a2dfc3cf4ed9d5dcfbb47fcb5a3a7851f3a2977e59e6acb45d3c9d27fa3f7ab"} Nov 02 20:04:57.208869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:57.208325 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:04:57.208869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:57.208383 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:04:57.238315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:57.237718 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:04:57.238315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:57.237781 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:04:57.723721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:57.723684 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:04:58.151304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.150494 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:58.208016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.207979 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:04:58.208191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.208030 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:04:58.243305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.236696 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:04:58.243305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.236757 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:04:58.301249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.300268 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-kube-api-access-gv9fh\" (UniqueName: \"kubernetes.io/projected/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-kube-api-access-gv9fh\") pod \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " Nov 02 20:04:58.301249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.300384 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-openshift-ca\") pod \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " Nov 02 20:04:58.301249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.300403 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-xks-ca\") pod \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\" (UID: \"f95f78e2-e9d7-450f-afb4-144cdaa5ab28\") " Nov 02 20:04:58.301249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.300962 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-openshift-ca" pod "f95f78e2-e9d7-450f-afb4-144cdaa5ab28" (UID: "f95f78e2-e9d7-450f-afb4-144cdaa5ab28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:04:58.302695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.302653 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-kube-api-access-gv9fh" pod "f95f78e2-e9d7-450f-afb4-144cdaa5ab28" (UID: "f95f78e2-e9d7-450f-afb4-144cdaa5ab28"). InnerVolumeSpecName "kube-api-access-gv9fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:04:58.307321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.305601 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-xks-ca" pod "f95f78e2-e9d7-450f-afb4-144cdaa5ab28" (UID: "f95f78e2-e9d7-450f-afb4-144cdaa5ab28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:04:58.401983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.401885 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:58.401983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.401918 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:58.401983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.401929 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9fh\" (UniqueName: \"kubernetes.io/projected/f95f78e2-e9d7-450f-afb4-144cdaa5ab28-kube-api-access-gv9fh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:04:58.824228 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.824190 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" event={"ID":"f95f78e2-e9d7-450f-afb4-144cdaa5ab28","Type":"ContainerDied","Data":"0fdafa8728c20d46a685ddfd8c087c8816776c0379cf5a3d36d2b8568ea72135"} Nov 02 20:04:58.824433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.824243 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fdafa8728c20d46a685ddfd8c087c8816776c0379cf5a3d36d2b8568ea72135" Nov 02 20:04:58.824433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:58.824333 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52" Nov 02 20:04:59.208564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:59.208480 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:04:59.208564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:59.208542 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:04:59.247305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:59.246761 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:04:59.247305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:04:59.246823 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:05:00.212370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:00.211638 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:05:00.212370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:00.211702 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:05:00.258306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:00.249397 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:05:00.258306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:00.249459 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:05:01.207779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:01.207743 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:05:01.207963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:01.207799 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:05:01.238306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:01.236661 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:05:01.238306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:01.236722 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:05:02.207843 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:02.207783 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:05:02.208391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:02.207950 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:05:02.252875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:02.252820 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:05:02.253227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:02.253195 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:05:03.208088 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:03.208044 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:05:03.208635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:03.208094 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 20:05:03.236704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:03.236655 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 02 20:05:03.236908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:03.236712 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 02 20:05:04.482033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:04.481994 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:05:04.482734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:04.482704 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:05:04.483707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:04.483687 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:05:04.609995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:04.609955 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:05:04.610704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:04.610668 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:05:04.611710 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:04.611688 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:05:07.722845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:07.722802 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:17.722652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:17.722608 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:22.695920 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:22.695874 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:05:27.722755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:27.722717 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:35.196563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.196529 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7"] Nov 02 20:05:35.198013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.197977 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:35.219852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.219826 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l"] Nov 02 20:05:35.220383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.220367 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f95f78e2-e9d7-450f-afb4-144cdaa5ab28" containerName="revoke-keys" Nov 02 20:05:35.220459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.220385 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95f78e2-e9d7-450f-afb4-144cdaa5ab28" containerName="revoke-keys" Nov 02 20:05:35.220548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.220494 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95f78e2-e9d7-450f-afb4-144cdaa5ab28" containerName="revoke-keys" Nov 02 20:05:35.225390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.225370 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.290755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.290722 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-plugins-config-volume\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.290960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.290778 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-routing-overlay-volume\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.290960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.290831 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ns4x\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-kube-api-access-9ns4x\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.291091 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.291042 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-provider-credentials\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.306339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.306311 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l"] Nov 02 20:05:35.391770 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.391733 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-provider-credentials\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.391945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.391889 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-plugins-config-volume\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.391945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.391929 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-routing-overlay-volume\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.392067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.391986 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ns4x\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-kube-api-access-9ns4x\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.392773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.392743 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-routing-overlay-volume\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.392888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.392842 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-plugins-config-volume\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.394236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.394217 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-provider-credentials\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.446152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.446118 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ns4x\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-kube-api-access-9ns4x\") pod \"payload-processing-external-model-cd9f598dd-4ds5l\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.541496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.541462 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:35.915521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:35.915486 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l"] Nov 02 20:05:35.917050 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:05:35.917028 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937b73c2_edcf_443b_b715_107ead2ef70f.slice/crio-e06a13ab71a28016f299398fb2c618e0404885d9a5222ec4fc019f691d475181 WatchSource:0}: Error finding container e06a13ab71a28016f299398fb2c618e0404885d9a5222ec4fc019f691d475181: Status 404 returned error can't find the container with id e06a13ab71a28016f299398fb2c618e0404885d9a5222ec4fc019f691d475181 Nov 02 20:05:36.177617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:36.177509 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" event={"ID":"937b73c2-edcf-443b-b715-107ead2ef70f","Type":"ContainerStarted","Data":"27bcd8825790a8bf302462e6e59f78d7f8256af13925e0184e7031537049e28a"} Nov 02 20:05:36.177617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:36.177575 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" event={"ID":"937b73c2-edcf-443b-b715-107ead2ef70f","Type":"ContainerStarted","Data":"e06a13ab71a28016f299398fb2c618e0404885d9a5222ec4fc019f691d475181"} Nov 02 20:05:36.177617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:36.177599 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:05:36.286793 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:36.286718 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" podStartSLOduration=1.286696922 podStartE2EDuration="1.286696922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:36.283640605 +0000 UTC m=+1034.800413190" watchObservedRunningTime="2026-11-02 20:05:36.286696922 +0000 UTC m=+1034.803469508" Nov 02 20:05:40.197832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.197792 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" containerID="cri-o://aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb" gracePeriod=25 Nov 02 20:05:40.483658 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.483633 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:05:40.552185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552150 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-routing-overlay-volume\") pod \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " Nov 02 20:05:40.552408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552276 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-plugins-config-volume\") pod \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " Nov 02 20:05:40.552408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552367 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-provider-credentials\") pod \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " Nov 02 20:05:40.552551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552414 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-kube-api-access-c6smb\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-kube-api-access-c6smb\") pod \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\" (UID: \"6b4b868a-a9a6-429a-be54-14a73a45f7a1\") " Nov 02 20:05:40.552639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552609 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-routing-overlay-volume" pod "6b4b868a-a9a6-429a-be54-14a73a45f7a1" (UID: "6b4b868a-a9a6-429a-be54-14a73a45f7a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:40.552639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552629 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-plugins-config-volume" pod "6b4b868a-a9a6-429a-be54-14a73a45f7a1" (UID: "6b4b868a-a9a6-429a-be54-14a73a45f7a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:05:40.552822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552784 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:40.552822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.552807 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b868a-a9a6-429a-be54-14a73a45f7a1-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:40.554651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.554626 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-provider-credentials" pod "6b4b868a-a9a6-429a-be54-14a73a45f7a1" (UID: "6b4b868a-a9a6-429a-be54-14a73a45f7a1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:40.554831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.554813 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-kube-api-access-c6smb" pod "6b4b868a-a9a6-429a-be54-14a73a45f7a1" (UID: "6b4b868a-a9a6-429a-be54-14a73a45f7a1"). InnerVolumeSpecName "kube-api-access-c6smb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:05:40.654185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.654135 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:40.654185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:40.654177 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6smb\" (UniqueName: \"kubernetes.io/projected/6b4b868a-a9a6-429a-be54-14a73a45f7a1-kube-api-access-c6smb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:05:41.212059 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.212025 5800 generic.go:356] "Generic (PLEG): container finished" podID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerID="aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb" exitCode=0 Nov 02 20:05:41.212528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.212070 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" Nov 02 20:05:41.212528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.212106 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" event={"ID":"6b4b868a-a9a6-429a-be54-14a73a45f7a1","Type":"ContainerDied","Data":"aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb"} Nov 02 20:05:41.212528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.212159 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7" event={"ID":"6b4b868a-a9a6-429a-be54-14a73a45f7a1","Type":"ContainerDied","Data":"c4d7b4261321b8716e1bbd28489b033d577a015b634ff89e8ea2005b6b412fad"} Nov 02 20:05:41.212528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.212180 5800 scope.go:122] "RemoveContainer" containerID="aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb" Nov 02 20:05:41.222749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.222732 5800 scope.go:122] "RemoveContainer" containerID="aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb" Nov 02 20:05:41.223051 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:41.223031 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb\": container with ID starting with aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb not found: ID does not exist" containerID="aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb" Nov 02 20:05:41.223106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.223059 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb"} err="failed to get container status \"aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb\": rpc error: code = NotFound desc = could not find container \"aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb\": container with ID starting with aebc1fe81a09c8e3f7bc9c86bc8a9da92b27bb9038bb65564fd47d38c2aaa1bb not found: ID does not exist" Nov 02 20:05:41.369363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.369324 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7"] Nov 02 20:05:41.371768 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.371738 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ccc49f6cb-mnpl7"] Nov 02 20:05:41.622414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:41.622385 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" path="/var/lib/kubelet/pods/6b4b868a-a9a6-429a-be54-14a73a45f7a1/volumes" Nov 02 20:05:47.187511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:47.187424 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:51.678885 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:51.678840 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2170d8c02f2: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:51.678885 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:51.678880 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:51.679386 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:51.678964 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials podName:bf500490-3634-4b1f-a950-7cc11ef7fe12 nodeName:}" failed. No retries permitted until 2026-11-02 20:05:52.178941428 +0000 UTC m=+1050.695714007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials") pod "payload-processing-external-model-74f4f7968-rjh4l" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12") : secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:52.184074 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:52.184040 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2170d8c02f2: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:52.184074 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:52.184069 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:52.184316 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:52.184135 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials podName:bf500490-3634-4b1f-a950-7cc11ef7fe12 nodeName:}" failed. No retries permitted until 2026-11-02 20:05:53.184118972 +0000 UTC m=+1051.700891535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials") pod "payload-processing-external-model-74f4f7968-rjh4l" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12") : secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:52.696077 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:52.696040 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:05:53.193841 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:53.193810 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2170d8c02f2: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:53.193841 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:53.193836 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:53.194079 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:53.193902 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials podName:bf500490-3634-4b1f-a950-7cc11ef7fe12 nodeName:}" failed. No retries permitted until 2026-11-02 20:05:55.193886 +0000 UTC m=+1053.710658576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials") pod "payload-processing-external-model-74f4f7968-rjh4l" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12") : secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:53.700745 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.700709 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h"] Nov 02 20:05:53.701322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.701305 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" Nov 02 20:05:53.701379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.701324 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" Nov 02 20:05:53.701439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.701429 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4b868a-a9a6-429a-be54-14a73a45f7a1" containerName="payload-processing" Nov 02 20:05:53.706690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.706669 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.757890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.757857 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h"] Nov 02 20:05:53.800169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.800134 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"kube-root-ca.crt\"" Nov 02 20:05:53.800344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.800194 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"distinct-w4-c25349-kserve-self-signed-certs\"" Nov 02 20:05:53.800344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.800204 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"openshift-service-ca.crt\"" Nov 02 20:05:53.801936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.801896 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/ad34c781-a25a-43ff-9ee2-39c875333cee-kube-api-access-ng78h\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.802066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.802001 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-dshm\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.802066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.802036 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c25349\"/\"default-dockercfg-hn4wb\"" Nov 02 20:05:53.802066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.802042 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34c781-a25a-43ff-9ee2-39c875333cee-tls-certs\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.802268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.802251 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-home\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.802396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.802377 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-tmp-dir\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.802457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.802430 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-model-cache\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904146 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-tmp-dir\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904202 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-model-cache\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904299 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/ad34c781-a25a-43ff-9ee2-39c875333cee-kube-api-access-ng78h\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904338 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-dshm\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904459 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34c781-a25a-43ff-9ee2-39c875333cee-tls-certs\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904571 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-tmp-dir\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904636 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-model-cache\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904638 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-home\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.904992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.904955 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-home\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.906780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.906755 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-dshm\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.907159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.907138 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34c781-a25a-43ff-9ee2-39c875333cee-tls-certs\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:53.957180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:53.957109 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/ad34c781-a25a-43ff-9ee2-39c875333cee-kube-api-access-ng78h\") pod \"distinct-w4-c25349-kserve-758df8b555-8lh7h\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:54.019662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:54.019633 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:54.398296 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:54.398257 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h"] Nov 02 20:05:54.399029 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:05:54.399003 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad34c781_a25a_43ff_9ee2_39c875333cee.slice/crio-4baac6784e0e32734ceb3b93b000b52b44ecbebfcbb9731c96b48e3c7bad4a1d WatchSource:0}: Error finding container 4baac6784e0e32734ceb3b93b000b52b44ecbebfcbb9731c96b48e3c7bad4a1d: Status 404 returned error can't find the container with id 4baac6784e0e32734ceb3b93b000b52b44ecbebfcbb9731c96b48e3c7bad4a1d Nov 02 20:05:55.220901 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:55.220863 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2170d8c02f2: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:55.220901 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:55.220892 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:55.221470 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:55.220951 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials podName:bf500490-3634-4b1f-a950-7cc11ef7fe12 nodeName:}" failed. No retries permitted until 2026-11-02 20:05:59.220931058 +0000 UTC m=+1057.737703620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials") pod "payload-processing-external-model-74f4f7968-rjh4l" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12") : secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:55.357942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:55.357905 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" event={"ID":"ad34c781-a25a-43ff-9ee2-39c875333cee","Type":"ContainerStarted","Data":"4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60"} Nov 02 20:05:55.357942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:55.357946 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" event={"ID":"ad34c781-a25a-43ff-9ee2-39c875333cee","Type":"ContainerStarted","Data":"4baac6784e0e32734ceb3b93b000b52b44ecbebfcbb9731c96b48e3c7bad4a1d"} Nov 02 20:05:55.469124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:55.469068 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podStartSLOduration=2.469054274 podStartE2EDuration="2.469054274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:05:55.466353673 +0000 UTC m=+1053.983126259" watchObservedRunningTime="2026-11-02 20:05:55.469054274 +0000 UTC m=+1053.985826859" Nov 02 20:05:56.020369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:56.020329 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:05:57.186109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:05:57.186073 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:05:59.269947 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:59.269904 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2170d8c02f2: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:59.269947 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:59.269932 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:05:59.270557 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:05:59.269996 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials podName:bf500490-3634-4b1f-a950-7cc11ef7fe12 nodeName:}" failed. No retries permitted until 2026-11-02 20:06:07.269981788 +0000 UTC m=+1065.786754355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials") pod "payload-processing-external-model-74f4f7968-rjh4l" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12") : secret "agc-cred-e2170d8c02f2" not found Nov 02 20:06:04.019986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:04.019945 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:06:04.032406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:04.032376 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:06:04.435339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:04.435242 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:06:06.224968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.224929 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l"] Nov 02 20:06:06.228067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.228027 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:06:06.254634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.254596 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4"] Nov 02 20:06:06.258587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.258565 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.307394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.307371 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b94811517373\"" Nov 02 20:06:06.307540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.307436 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0188ccd53d43\"" Nov 02 20:06:06.324187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.324157 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4"] Nov 02 20:06:06.356495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.356461 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-routing-overlay-volume\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.356664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.356601 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.356664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.356639 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-plugins-config-volume\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.356819 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.356779 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqnm\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-kube-api-access-7kqnm\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.457688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.457650 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-routing-overlay-volume\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.457856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.457727 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.457856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.457745 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-plugins-config-volume\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.458004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.457858 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqnm\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-kube-api-access-7kqnm\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.458409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.458383 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-routing-overlay-volume\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.458505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.458487 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-plugins-config-volume\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.460275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.460256 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.511056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.511021 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqnm\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-kube-api-access-7kqnm\") pod \"payload-processing-external-model-6d54dc9bd-5rrc4\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.573387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.573352 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:06.956499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:06.956471 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4"] Nov 02 20:06:06.957446 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:06:06.957413 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d008eda_79f5_4bce_a0e6_40ce42d67ee5.slice/crio-9623ef54c7a942a286b8418c41bd0fccec806b25148e58e26068b29fa7447f2a WatchSource:0}: Error finding container 9623ef54c7a942a286b8418c41bd0fccec806b25148e58e26068b29fa7447f2a: Status 404 returned error can't find the container with id 9623ef54c7a942a286b8418c41bd0fccec806b25148e58e26068b29fa7447f2a Nov 02 20:06:07.368325 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:07.368289 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2170d8c02f2: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:06:07.368325 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:07.368317 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:06:07.368752 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:07.368370 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials podName:bf500490-3634-4b1f-a950-7cc11ef7fe12 nodeName:}" failed. No retries permitted until 2026-11-02 20:06:23.368356282 +0000 UTC m=+1081.885128845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials") pod "payload-processing-external-model-74f4f7968-rjh4l" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12") : secret "agc-cred-e2170d8c02f2" not found Nov 02 20:06:07.446031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:07.445995 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" event={"ID":"1d008eda-79f5-4bce-a0e6-40ce42d67ee5","Type":"ContainerStarted","Data":"f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81"} Nov 02 20:06:07.446227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:07.446051 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" event={"ID":"1d008eda-79f5-4bce-a0e6-40ce42d67ee5","Type":"ContainerStarted","Data":"9623ef54c7a942a286b8418c41bd0fccec806b25148e58e26068b29fa7447f2a"} Nov 02 20:06:07.446227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:07.446184 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:07.553143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:07.553088 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" podStartSLOduration=1.5530727469999999 podStartE2EDuration="1.553072747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:07.5513418 +0000 UTC m=+1066.068114386" watchObservedRunningTime="2026-11-02 20:06:07.553072747 +0000 UTC m=+1066.069845489" Nov 02 20:06:11.226802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.226755 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" containerName="payload-processing" containerID="cri-o://27bcd8825790a8bf302462e6e59f78d7f8256af13925e0184e7031537049e28a" gracePeriod=25 Nov 02 20:06:11.476154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.476112 5800 generic.go:356] "Generic (PLEG): container finished" podID="937b73c2-edcf-443b-b715-107ead2ef70f" containerID="27bcd8825790a8bf302462e6e59f78d7f8256af13925e0184e7031537049e28a" exitCode=0 Nov 02 20:06:11.476359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.476181 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" event={"ID":"937b73c2-edcf-443b-b715-107ead2ef70f","Type":"ContainerDied","Data":"27bcd8825790a8bf302462e6e59f78d7f8256af13925e0184e7031537049e28a"} Nov 02 20:06:11.518634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.518602 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:06:11.617396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.617356 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-kube-api-access-9ns4x\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-kube-api-access-9ns4x\") pod \"937b73c2-edcf-443b-b715-107ead2ef70f\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " Nov 02 20:06:11.617602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.617502 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-provider-credentials\") pod \"937b73c2-edcf-443b-b715-107ead2ef70f\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " Nov 02 20:06:11.617602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.617551 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-plugins-config-volume\") pod \"937b73c2-edcf-443b-b715-107ead2ef70f\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " Nov 02 20:06:11.617722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.617621 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-routing-overlay-volume\") pod \"937b73c2-edcf-443b-b715-107ead2ef70f\" (UID: \"937b73c2-edcf-443b-b715-107ead2ef70f\") " Nov 02 20:06:11.618027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.617996 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-plugins-config-volume" pod "937b73c2-edcf-443b-b715-107ead2ef70f" (UID: "937b73c2-edcf-443b-b715-107ead2ef70f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:11.618027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.618005 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-routing-overlay-volume" pod "937b73c2-edcf-443b-b715-107ead2ef70f" (UID: "937b73c2-edcf-443b-b715-107ead2ef70f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:11.618244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.618214 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:11.618379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.618252 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/937b73c2-edcf-443b-b715-107ead2ef70f-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:11.619941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.619918 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-kube-api-access-9ns4x" pod "937b73c2-edcf-443b-b715-107ead2ef70f" (UID: "937b73c2-edcf-443b-b715-107ead2ef70f"). InnerVolumeSpecName "kube-api-access-9ns4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:11.620207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.620187 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-provider-credentials" pod "937b73c2-edcf-443b-b715-107ead2ef70f" (UID: "937b73c2-edcf-443b-b715-107ead2ef70f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:11.719602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.719561 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ns4x\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-kube-api-access-9ns4x\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:11.719602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:11.719600 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/937b73c2-edcf-443b-b715-107ead2ef70f-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:12.484558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:12.484510 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" event={"ID":"937b73c2-edcf-443b-b715-107ead2ef70f","Type":"ContainerDied","Data":"e06a13ab71a28016f299398fb2c618e0404885d9a5222ec4fc019f691d475181"} Nov 02 20:06:12.484558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:12.484544 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l" Nov 02 20:06:12.484558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:12.484572 5800 scope.go:122] "RemoveContainer" containerID="27bcd8825790a8bf302462e6e59f78d7f8256af13925e0184e7031537049e28a" Nov 02 20:06:12.661662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:12.661623 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l"] Nov 02 20:06:12.693116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:12.693081 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-4ds5l"] Nov 02 20:06:13.625943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:13.625908 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" path="/var/lib/kubelet/pods/937b73c2-edcf-443b-b715-107ead2ef70f/volumes" Nov 02 20:06:18.455492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:18.455456 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:06:18.612332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:18.612297 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l"] Nov 02 20:06:19.201983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.201941 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997"] Nov 02 20:06:19.202494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.202479 5800 cpu_manager.go:414] "containerMap: removing container" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" containerName="payload-processing" Nov 02 20:06:19.202546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.202496 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" containerName="payload-processing" Nov 02 20:06:19.202609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.202600 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="937b73c2-edcf-443b-b715-107ead2ef70f" containerName="payload-processing" Nov 02 20:06:19.206863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.206847 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.270887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.270842 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.270887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.270888 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00022ea6-6d93-402b-8043-b8b1e013e185-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.271088 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.270970 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.271088 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.271000 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j89jk\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-kube-api-access-j89jk\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.271211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.271079 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.271211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.271121 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.271211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.271157 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00022ea6-6d93-402b-8043-b8b1e013e185-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.271355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.271269 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.271355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.271336 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.313139 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.313110 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997"] Nov 02 20:06:19.372191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372158 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372216 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372256 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372274 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00022ea6-6d93-402b-8043-b8b1e013e185-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372344 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372377 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89jk\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-kube-api-access-j89jk\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372440 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372480 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372534 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00022ea6-6d93-402b-8043-b8b1e013e185-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372613 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.372922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372900 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.373018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.372934 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.373076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.373032 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.373346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.373324 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00022ea6-6d93-402b-8043-b8b1e013e185-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.374764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.374733 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.374884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.374845 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00022ea6-6d93-402b-8043-b8b1e013e185-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.425564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.425534 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.425754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.425731 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89jk\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-kube-api-access-j89jk\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.520517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.520483 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:19.938513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.938486 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997"] Nov 02 20:06:19.940267 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:06:19.940239 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00022ea6_6d93_402b_8043_b8b1e013e185.slice/crio-1f39c563d9c10c44bf8fc35bbcfda9ab3776ddba2c352b9ff9383864b6fe9415 WatchSource:0}: Error finding container 1f39c563d9c10c44bf8fc35bbcfda9ab3776ddba2c352b9ff9383864b6fe9415: Status 404 returned error can't find the container with id 1f39c563d9c10c44bf8fc35bbcfda9ab3776ddba2c352b9ff9383864b6fe9415 Nov 02 20:06:19.942688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.942652 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:06:19.942786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.942720 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:06:19.942786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:19.942747 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:06:20.571478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:20.571439 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" event={"ID":"00022ea6-6d93-402b-8043-b8b1e013e185","Type":"ContainerStarted","Data":"c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2"} Nov 02 20:06:20.571934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:20.571498 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" event={"ID":"00022ea6-6d93-402b-8043-b8b1e013e185","Type":"ContainerStarted","Data":"1f39c563d9c10c44bf8fc35bbcfda9ab3776ddba2c352b9ff9383864b6fe9415"} Nov 02 20:06:20.789332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:20.789258 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podStartSLOduration=1.789242243 podStartE2EDuration="1.789242243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:20.741017535 +0000 UTC m=+1079.257790121" watchObservedRunningTime="2026-11-02 20:06:20.789242243 +0000 UTC m=+1079.306014902" Nov 02 20:06:21.520765 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:21.520732 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:21.522385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:21.522356 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:06:21.522521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:21.522408 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 20:06:21.920869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:21.920779 5800 scope.go:122] "RemoveContainer" containerID="0653ee5b2ce397c8de46b108a8e8903ba04396e230fd11fa11fda26e54cc11ee" Nov 02 20:06:22.520934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:22.520890 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:06:22.521118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:22.520956 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 20:06:22.695801 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:22.695767 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:06:23.417924 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:23.417890 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e2170d8c02f2: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:06:23.417924 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:23.417920 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l: secret "agc-cred-e2170d8c02f2" not found Nov 02 20:06:23.418394 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:23.417987 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials podName:bf500490-3634-4b1f-a950-7cc11ef7fe12 nodeName:}" failed. No retries permitted until 2026-11-02 20:06:55.417965567 +0000 UTC m=+1113.934738143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials") pod "payload-processing-external-model-74f4f7968-rjh4l" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12") : secret "agc-cred-e2170d8c02f2" not found Nov 02 20:06:23.521314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:23.521266 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:06:23.521508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:23.521327 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 20:06:23.612791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:23.612756 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" podUID="bf500490-3634-4b1f-a950-7cc11ef7fe12" containerName="payload-processing" containerID="cri-o://10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae" gracePeriod=25 Nov 02 20:06:23.954156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:23.954125 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:06:24.031307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.029707 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-plugins-config-volume\") pod \"bf500490-3634-4b1f-a950-7cc11ef7fe12\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " Nov 02 20:06:24.031307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.029801 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials\") pod \"bf500490-3634-4b1f-a950-7cc11ef7fe12\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " Nov 02 20:06:24.031307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.029865 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-kube-api-access-vgmfd\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-kube-api-access-vgmfd\") pod \"bf500490-3634-4b1f-a950-7cc11ef7fe12\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " Nov 02 20:06:24.031307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.029928 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-routing-overlay-volume\") pod \"bf500490-3634-4b1f-a950-7cc11ef7fe12\" (UID: \"bf500490-3634-4b1f-a950-7cc11ef7fe12\") " Nov 02 20:06:24.050611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.050567 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-routing-overlay-volume" pod "bf500490-3634-4b1f-a950-7cc11ef7fe12" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:24.051706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.051670 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-plugins-config-volume" pod "bf500490-3634-4b1f-a950-7cc11ef7fe12" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:06:24.053041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.053015 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-kube-api-access-vgmfd" pod "bf500490-3634-4b1f-a950-7cc11ef7fe12" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12"). InnerVolumeSpecName "kube-api-access-vgmfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:24.053148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.053035 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials" pod "bf500490-3634-4b1f-a950-7cc11ef7fe12" (UID: "bf500490-3634-4b1f-a950-7cc11ef7fe12"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:06:24.150148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.150103 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:24.150930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.150686 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:24.150930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.150725 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgmfd\" (UniqueName: \"kubernetes.io/projected/bf500490-3634-4b1f-a950-7cc11ef7fe12-kube-api-access-vgmfd\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:24.150930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.150744 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf500490-3634-4b1f-a950-7cc11ef7fe12-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:06:24.521105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.521070 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:06:24.521601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.521130 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 20:06:24.600325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.600275 5800 generic.go:356] "Generic (PLEG): container finished" podID="bf500490-3634-4b1f-a950-7cc11ef7fe12" containerID="10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae" exitCode=0 Nov 02 20:06:24.600492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.600342 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" Nov 02 20:06:24.600492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.600374 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" event={"ID":"bf500490-3634-4b1f-a950-7cc11ef7fe12","Type":"ContainerDied","Data":"10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae"} Nov 02 20:06:24.600492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.600455 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l" event={"ID":"bf500490-3634-4b1f-a950-7cc11ef7fe12","Type":"ContainerDied","Data":"757476f82ad7db328f25ec5b087fd4d6f97b3e3acfc0a165382b9a21be270b3d"} Nov 02 20:06:24.600492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.600482 5800 scope.go:122] "RemoveContainer" containerID="10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae" Nov 02 20:06:24.612497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.612476 5800 scope.go:122] "RemoveContainer" containerID="10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae" Nov 02 20:06:24.612776 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:24.612758 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae\": container with ID starting with 10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae not found: ID does not exist" containerID="10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae" Nov 02 20:06:24.612829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.612786 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae"} err="failed to get container status \"10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae\": rpc error: code = NotFound desc = could not find container \"10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae\": container with ID starting with 10b019c0d439bccfc56fdc441ea6c1ef1d89787b3c0142708992c5c4f43709ae not found: ID does not exist" Nov 02 20:06:24.760094 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.760063 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l"] Nov 02 20:06:24.762990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:24.762963 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-74f4f7968-rjh4l"] Nov 02 20:06:25.521554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:25.521515 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:06:25.521984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:25.521575 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 20:06:25.623128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:25.623094 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf500490-3634-4b1f-a950-7cc11ef7fe12" path="/var/lib/kubelet/pods/bf500490-3634-4b1f-a950-7cc11ef7fe12/volumes" Nov 02 20:06:26.521259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:26.521222 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 20:06:26.521448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:26.521306 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 20:06:27.525065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:27.525034 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:27.525678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:27.525654 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:27.526383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:27.526355 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:06:51.460724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.460689 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5"] Nov 02 20:06:51.461229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.461216 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bf500490-3634-4b1f-a950-7cc11ef7fe12" containerName="payload-processing" Nov 02 20:06:51.461300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.461230 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf500490-3634-4b1f-a950-7cc11ef7fe12" containerName="payload-processing" Nov 02 20:06:51.461351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.461341 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf500490-3634-4b1f-a950-7cc11ef7fe12" containerName="payload-processing" Nov 02 20:06:51.464775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.464759 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.525020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.524979 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-model-cache\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.525020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.525021 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqttc\" (UniqueName: \"kubernetes.io/projected/bf273e63-6dd2-49ea-9ffe-dca9a00de929-kube-api-access-dqttc\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.525259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.525125 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tmp-dir\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.525259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.525172 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tls-certs\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.525373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.525356 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-home\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.525420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.525398 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-dshm\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.556334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.556300 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5"] Nov 02 20:06:51.561267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.561243 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-426941\"/\"distinct-w2-426941-kserve-self-signed-certs\"" Nov 02 20:06:51.627542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.627505 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-model-cache\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.627894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.627867 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqttc\" (UniqueName: \"kubernetes.io/projected/bf273e63-6dd2-49ea-9ffe-dca9a00de929-kube-api-access-dqttc\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.628063 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.628044 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tmp-dir\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.628185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.628166 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tls-certs\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.628350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.628333 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-home\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.628455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.628378 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-dshm\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.633520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.629534 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-model-cache\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.633520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.629801 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-home\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.633520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.630130 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tmp-dir\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.633951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.633601 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tls-certs\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.634760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.634709 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-dshm\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.683077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.683040 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqttc\" (UniqueName: \"kubernetes.io/projected/bf273e63-6dd2-49ea-9ffe-dca9a00de929-kube-api-access-dqttc\") pod \"distinct-w2-426941-kserve-78d6f46875-qzdm5\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:51.776933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:51.776898 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:06:52.161886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:52.161860 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5"] Nov 02 20:06:52.162793 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:06:52.162769 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf273e63_6dd2_49ea_9ffe_dca9a00de929.slice/crio-ed96648eef74e8abdab5eb4d313dce812101d6e8b6c3354f8f0b052f45de033d WatchSource:0}: Error finding container ed96648eef74e8abdab5eb4d313dce812101d6e8b6c3354f8f0b052f45de033d: Status 404 returned error can't find the container with id ed96648eef74e8abdab5eb4d313dce812101d6e8b6c3354f8f0b052f45de033d Nov 02 20:06:52.695719 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:06:52.695679 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:06:52.843166 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:52.843131 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" event={"ID":"bf273e63-6dd2-49ea-9ffe-dca9a00de929","Type":"ContainerStarted","Data":"399abbd08706a5245015ada21011cd7b0b212f5b7b7fe46c6aa9d94d15ad962b"} Nov 02 20:06:52.843375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:52.843182 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" event={"ID":"bf273e63-6dd2-49ea-9ffe-dca9a00de929","Type":"ContainerStarted","Data":"ed96648eef74e8abdab5eb4d313dce812101d6e8b6c3354f8f0b052f45de033d"} Nov 02 20:06:52.950922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:52.950818 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podStartSLOduration=1.950804719 podStartE2EDuration="1.950804719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:06:52.949159741 +0000 UTC m=+1111.465932338" watchObservedRunningTime="2026-11-02 20:06:52.950804719 +0000 UTC m=+1111.467577320" Nov 02 20:06:53.781212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:06:53.778249 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:07:01.777585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:01.777555 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:07:01.791253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:01.791221 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:07:01.947248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:01.947213 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:07:09.631084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.631051 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c"] Nov 02 20:07:09.635935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.635910 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.684828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.684798 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bbee67-openshift-default-dockercfg-lw7rw\"" Nov 02 20:07:09.735106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.735074 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c"] Nov 02 20:07:09.746207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746179 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746263 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746390 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746427 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746454 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26e053c-6975-4224-ac62-9a054e216d84-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746499 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746543 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746561 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmzb\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-kube-api-access-vlmzb\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.746662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.746635 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26e053c-6975-4224-ac62-9a054e216d84-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.849734 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.849845 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.849914 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.849938 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.849979 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26e053c-6975-4224-ac62-9a054e216d84-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.850005 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.850042 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.850066 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmzb\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-kube-api-access-vlmzb\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.850105 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26e053c-6975-4224-ac62-9a054e216d84-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.851044 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26e053c-6975-4224-ac62-9a054e216d84-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.852075 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.852344 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.852603 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.853307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.852841 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.858728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.858681 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26e053c-6975-4224-ac62-9a054e216d84-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.858728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.858711 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.905954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.905877 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.906098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.906017 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmzb\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-kube-api-access-vlmzb\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:09.951344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:09.951313 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:10.382580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:10.382549 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c"] Nov 02 20:07:10.383862 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:07:10.383827 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26e053c_6975_4224_ac62_9a054e216d84.slice/crio-90b15fdfe00c28f3be887d3fc842dfae069fe5b98463d3eb088bb4b66021a4b8 WatchSource:0}: Error finding container 90b15fdfe00c28f3be887d3fc842dfae069fe5b98463d3eb088bb4b66021a4b8: Status 404 returned error can't find the container with id 90b15fdfe00c28f3be887d3fc842dfae069fe5b98463d3eb088bb4b66021a4b8 Nov 02 20:07:10.386117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:10.386083 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:07:10.386190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:10.386150 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:07:10.386190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:10.386178 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:07:10.999183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:10.999142 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" event={"ID":"d26e053c-6975-4224-ac62-9a054e216d84","Type":"ContainerStarted","Data":"47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e"} Nov 02 20:07:10.999639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:10.999196 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" event={"ID":"d26e053c-6975-4224-ac62-9a054e216d84","Type":"ContainerStarted","Data":"90b15fdfe00c28f3be887d3fc842dfae069fe5b98463d3eb088bb4b66021a4b8"} Nov 02 20:07:11.162081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:11.162004 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podStartSLOduration=2.161981483 podStartE2EDuration="2.161981483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:11.10824393 +0000 UTC m=+1129.625016515" watchObservedRunningTime="2026-11-02 20:07:11.161981483 +0000 UTC m=+1129.678754046" Nov 02 20:07:11.954800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:11.954510 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:11.954800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:11.954674 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 20:07:11.954800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:11.954736 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 20:07:12.952462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:12.952424 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 20:07:12.953060 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:12.953019 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 20:07:13.952744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:13.952706 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 20:07:13.953262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:13.952766 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 20:07:14.952704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:14.952655 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 20:07:14.952895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:14.952718 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 20:07:15.951998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:15.951960 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 20:07:15.952211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:15.952026 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 20:07:16.952203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:16.952168 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 20:07:16.952794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:16.952226 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 20:07:17.956657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:17.956628 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:17.957111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:17.957099 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:17.957794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:17.957774 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:07:22.700686 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:07:22.697450 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:07:31.430156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.430117 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xx7zt-m4wgl"] Nov 02 20:07:31.436210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.436184 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.484291 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.484239 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xx7zt-m4wgl"] Nov 02 20:07:31.535636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.535600 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8m2b\" (UniqueName: \"kubernetes.io/projected/b4e9df5c-b0ff-456e-b298-d2115c99711d-kube-api-access-h8m2b\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.535636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.535643 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-trusted-ca-bundle\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.535897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.535799 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-service-ca-bundle\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.536013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.535995 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4e9df5c-b0ff-456e-b298-d2115c99711d-archives-path\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.637208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.637171 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4e9df5c-b0ff-456e-b298-d2115c99711d-archives-path\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.637438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.637255 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8m2b\" (UniqueName: \"kubernetes.io/projected/b4e9df5c-b0ff-456e-b298-d2115c99711d-kube-api-access-h8m2b\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.637438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.637303 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-trusted-ca-bundle\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.637562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.637466 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-service-ca-bundle\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.637619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.637600 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4e9df5c-b0ff-456e-b298-d2115c99711d-archives-path\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.638001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.637984 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-service-ca-bundle\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.638081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.638065 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-trusted-ca-bundle\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.689832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.689767 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8m2b\" (UniqueName: \"kubernetes.io/projected/b4e9df5c-b0ff-456e-b298-d2115c99711d-kube-api-access-h8m2b\") pod \"periodic-gathering-xx7zt-m4wgl\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:31.748408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:31.748377 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:07:32.115118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:32.115089 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xx7zt-m4wgl"] Nov 02 20:07:32.115750 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:07:32.115714 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e9df5c_b0ff_456e_b298_d2115c99711d.slice/crio-c9d73aa1693c26302048aa61af725148dba052a1d3c5120737eb2576fb5880ae WatchSource:0}: Error finding container c9d73aa1693c26302048aa61af725148dba052a1d3c5120737eb2576fb5880ae: Status 404 returned error can't find the container with id c9d73aa1693c26302048aa61af725148dba052a1d3c5120737eb2576fb5880ae Nov 02 20:07:32.183809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:32.183781 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" event={"ID":"b4e9df5c-b0ff-456e-b298-d2115c99711d","Type":"ContainerStarted","Data":"c9d73aa1693c26302048aa61af725148dba052a1d3c5120737eb2576fb5880ae"} Nov 02 20:07:33.194796 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.194757 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" event={"ID":"b4e9df5c-b0ff-456e-b298-d2115c99711d","Type":"ContainerStarted","Data":"0f51a9bbcb206f991f802964adf82faaffe1cf4fbd9b394555e2e5934fa95cd1"} Nov 02 20:07:33.196312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.196272 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22"] Nov 02 20:07:33.200343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.200322 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.255383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255345 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.255555 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255393 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.255555 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255410 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwstv\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-kube-api-access-zwstv\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.255682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255643 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.255739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255689 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.255800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255782 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.255862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255847 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.256007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.255991 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.256050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.256032 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.296616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.296581 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22"] Nov 02 20:07:33.339363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.339294 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" podStartSLOduration=2.339260166 podStartE2EDuration="2.339260166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:33.337468139 +0000 UTC m=+1151.854240732" watchObservedRunningTime="2026-11-02 20:07:33.339260166 +0000 UTC m=+1151.856032752" Nov 02 20:07:33.356974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.356937 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.356985 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357001 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwstv\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-kube-api-access-zwstv\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357050 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357070 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357108 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357152 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357210 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357257 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357573 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357652 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.357861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.357835 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.358024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.358002 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.358092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.358010 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.359494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.359471 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.359750 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.359731 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.409773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.409742 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.409943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.409832 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwstv\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-kube-api-access-zwstv\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.515696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.515658 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:33.937682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.937644 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22"] Nov 02 20:07:33.938094 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:07:33.938070 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ca928f_ece7_4c63_aae8_3aa97f7becd7.slice/crio-064163519a3ef9417a93632cdced1eda05cb5f6ca12f37709fb1e0ca4314f514 WatchSource:0}: Error finding container 064163519a3ef9417a93632cdced1eda05cb5f6ca12f37709fb1e0ca4314f514: Status 404 returned error can't find the container with id 064163519a3ef9417a93632cdced1eda05cb5f6ca12f37709fb1e0ca4314f514 Nov 02 20:07:33.941412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.941381 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:07:33.941503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.941443 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:07:33.941503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:33.941468 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:07:34.204958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:34.204848 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" event={"ID":"e5ca928f-ece7-4c63-aae8-3aa97f7becd7","Type":"ContainerStarted","Data":"33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6"} Nov 02 20:07:34.204958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:34.204904 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" event={"ID":"e5ca928f-ece7-4c63-aae8-3aa97f7becd7","Type":"ContainerStarted","Data":"064163519a3ef9417a93632cdced1eda05cb5f6ca12f37709fb1e0ca4314f514"} Nov 02 20:07:34.315987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:34.315930 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podStartSLOduration=1.315914984 podStartE2EDuration="1.315914984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:07:34.312699803 +0000 UTC m=+1152.829472388" watchObservedRunningTime="2026-11-02 20:07:34.315914984 +0000 UTC m=+1152.832687568" Nov 02 20:07:34.516206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:34.516176 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:35.518173 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:35.518136 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:07:35.518586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:35.518196 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:07:36.516035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:36.516001 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:07:36.516295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:36.516261 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:07:37.516264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:37.516214 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:07:37.516674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:37.516271 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:07:38.517035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:38.516997 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:07:38.517460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:38.517060 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:07:39.516304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:39.516252 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:07:39.516492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:39.516323 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:07:40.516209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:40.516170 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 02 20:07:40.516633 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:40.516233 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 02 20:07:41.520110 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:41.520077 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:41.520562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:41.520501 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:41.521115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:07:41.521094 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:07:52.696054 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:07:52.696017 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:08:00.762362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.761007 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq"] Nov 02 20:08:00.797057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.797023 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:00.817801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.817763 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq"] Nov 02 20:08:00.871393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.868668 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:08:00.871393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.869501 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:08:00.893201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.893157 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:08:00.893897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.893646 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:08:00.944170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.944130 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:00.946321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.946276 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:00.946607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:00.946589 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6g6\" (UniqueName: \"kubernetes.io/projected/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-kube-api-access-nd6g6\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.050229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.050126 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.051152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.051119 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.051432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.051410 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6g6\" (UniqueName: \"kubernetes.io/projected/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-kube-api-access-nd6g6\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.056890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.056861 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.062717 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.062692 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.104591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.104559 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6g6\" (UniqueName: \"kubernetes.io/projected/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-kube-api-access-nd6g6\") pod \"maas-api-revoke-sub-e9bbf8e8-748zq\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.116448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.116417 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:01.555208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:01.555168 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq"] Nov 02 20:08:01.561750 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:01.561705 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0164d53_b5f8_4acc_b097_5a40a1ca4f71.slice/crio-507bafc9d01463b0d8094f1861310fca4b48e7094c467c145e77dd2f7e3889c1 WatchSource:0}: Error finding container 507bafc9d01463b0d8094f1861310fca4b48e7094c467c145e77dd2f7e3889c1: Status 404 returned error can't find the container with id 507bafc9d01463b0d8094f1861310fca4b48e7094c467c145e77dd2f7e3889c1 Nov 02 20:08:02.494929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:02.494891 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" event={"ID":"f0164d53-b5f8-4acc-b097-5a40a1ca4f71","Type":"ContainerStarted","Data":"490a1925ee6462ae989a2aae2b8168ac5e784a56a70f451c2f1446424b5d1dac"} Nov 02 20:08:02.495431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:02.494948 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" event={"ID":"f0164d53-b5f8-4acc-b097-5a40a1ca4f71","Type":"ContainerStarted","Data":"507bafc9d01463b0d8094f1861310fca4b48e7094c467c145e77dd2f7e3889c1"} Nov 02 20:08:02.612703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:02.612640 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" podStartSLOduration=2.612626246 podStartE2EDuration="2.612626246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:02.610171441 +0000 UTC m=+1181.126944028" watchObservedRunningTime="2026-11-02 20:08:02.612626246 +0000 UTC m=+1181.129398864" Nov 02 20:08:03.528341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:03.528305 5800 generic.go:356] "Generic (PLEG): container finished" podID="f0164d53-b5f8-4acc-b097-5a40a1ca4f71" containerID="490a1925ee6462ae989a2aae2b8168ac5e784a56a70f451c2f1446424b5d1dac" exitCode=0 Nov 02 20:08:03.528826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:03.528349 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" event={"ID":"f0164d53-b5f8-4acc-b097-5a40a1ca4f71","Type":"ContainerDied","Data":"490a1925ee6462ae989a2aae2b8168ac5e784a56a70f451c2f1446424b5d1dac"} Nov 02 20:08:04.684947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.684919 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:04.742007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.741976 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-kube-api-access-nd6g6\" (UniqueName: \"kubernetes.io/projected/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-kube-api-access-nd6g6\") pod \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " Nov 02 20:08:04.742159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.742058 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-xks-ca\") pod \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " Nov 02 20:08:04.742159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.742094 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-openshift-ca\") pod \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\" (UID: \"f0164d53-b5f8-4acc-b097-5a40a1ca4f71\") " Nov 02 20:08:04.742524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.742501 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-openshift-ca" pod "f0164d53-b5f8-4acc-b097-5a40a1ca4f71" (UID: "f0164d53-b5f8-4acc-b097-5a40a1ca4f71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:04.744170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.744146 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-xks-ca" pod "f0164d53-b5f8-4acc-b097-5a40a1ca4f71" (UID: "f0164d53-b5f8-4acc-b097-5a40a1ca4f71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:04.744269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.744197 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-kube-api-access-nd6g6" pod "f0164d53-b5f8-4acc-b097-5a40a1ca4f71" (UID: "f0164d53-b5f8-4acc-b097-5a40a1ca4f71"). InnerVolumeSpecName "kube-api-access-nd6g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:04.843845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.843758 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:04.843845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.843791 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:04.843845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:04.843805 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd6g6\" (UniqueName: \"kubernetes.io/projected/f0164d53-b5f8-4acc-b097-5a40a1ca4f71-kube-api-access-nd6g6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:05.541731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:05.541693 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" event={"ID":"f0164d53-b5f8-4acc-b097-5a40a1ca4f71","Type":"ContainerDied","Data":"507bafc9d01463b0d8094f1861310fca4b48e7094c467c145e77dd2f7e3889c1"} Nov 02 20:08:05.541908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:05.541750 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="507bafc9d01463b0d8094f1861310fca4b48e7094c467c145e77dd2f7e3889c1" Nov 02 20:08:05.541908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:05.541705 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq" Nov 02 20:08:17.614244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.614193 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr"] Nov 02 20:08:17.615388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.615362 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f0164d53-b5f8-4acc-b097-5a40a1ca4f71" containerName="revoke-keys" Nov 02 20:08:17.615388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.615387 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0164d53-b5f8-4acc-b097-5a40a1ca4f71" containerName="revoke-keys" Nov 02 20:08:17.615664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.615646 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0164d53-b5f8-4acc-b097-5a40a1ca4f71" containerName="revoke-keys" Nov 02 20:08:17.619843 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.619820 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.642238 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.642211 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.642382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.642321 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.642446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.642419 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwwk\" (UniqueName: \"kubernetes.io/projected/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-kube-api-access-xcwwk\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.669067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.669037 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr"] Nov 02 20:08:17.710464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.710421 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:08:17.710464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.710433 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:08:17.710673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.710480 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:08:17.715149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.715130 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:08:17.743311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.743271 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwwk\" (UniqueName: \"kubernetes.io/projected/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-kube-api-access-xcwwk\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.743481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.743388 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.743559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.743539 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.744191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.744168 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.745879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.745860 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.797526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.797498 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwwk\" (UniqueName: \"kubernetes.io/projected/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-kube-api-access-xcwwk\") pod \"maas-api-revoke-sub-41ad198a-xspfr\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:17.936776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:17.936695 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:18.305874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:18.305841 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr"] Nov 02 20:08:18.306639 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:18.306611 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8943ff71_b8e8_437c_ab45_7bb85c9fd6a9.slice/crio-6332c116ff2a153f25c774695bfd529894ca674dfea808fb393984a869ce5547 WatchSource:0}: Error finding container 6332c116ff2a153f25c774695bfd529894ca674dfea808fb393984a869ce5547: Status 404 returned error can't find the container with id 6332c116ff2a153f25c774695bfd529894ca674dfea808fb393984a869ce5547 Nov 02 20:08:18.714748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:18.714706 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" event={"ID":"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9","Type":"ContainerStarted","Data":"6332c116ff2a153f25c774695bfd529894ca674dfea808fb393984a869ce5547"} Nov 02 20:08:19.723014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:19.722927 5800 generic.go:356] "Generic (PLEG): container finished" podID="8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" containerID="dcff7e5dd4e89ca4a257fd90b0fb0dbdaf047986313378cdcd00472a1a679c6b" exitCode=0 Nov 02 20:08:19.723404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:19.723008 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" event={"ID":"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9","Type":"ContainerDied","Data":"dcff7e5dd4e89ca4a257fd90b0fb0dbdaf047986313378cdcd00472a1a679c6b"} Nov 02 20:08:20.906588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:20.906562 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:20.984036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:20.983958 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-xks-ca\") pod \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " Nov 02 20:08:20.984187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:20.984107 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-kube-api-access-xcwwk\" (UniqueName: \"kubernetes.io/projected/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-kube-api-access-xcwwk\") pod \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " Nov 02 20:08:20.984187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:20.984156 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-openshift-ca\") pod \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\" (UID: \"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9\") " Nov 02 20:08:20.984638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:20.984611 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-openshift-ca" pod "8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" (UID: "8943ff71-b8e8-437c-ab45-7bb85c9fd6a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:20.986206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:20.986181 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-kube-api-access-xcwwk" pod "8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" (UID: "8943ff71-b8e8-437c-ab45-7bb85c9fd6a9"). InnerVolumeSpecName "kube-api-access-xcwwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:20.986329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:20.986202 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-xks-ca" pod "8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" (UID: "8943ff71-b8e8-437c-ab45-7bb85c9fd6a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:21.085500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.085460 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:21.085500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.085493 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcwwk\" (UniqueName: \"kubernetes.io/projected/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-kube-api-access-xcwwk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:21.085500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.085503 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:21.218521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.218480 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr"] Nov 02 20:08:21.219428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.219407 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" containerName="revoke-keys" Nov 02 20:08:21.219547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.219430 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" containerName="revoke-keys" Nov 02 20:08:21.219616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.219604 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" containerName="revoke-keys" Nov 02 20:08:21.223248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.223227 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.275924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.275886 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr"] Nov 02 20:08:21.288373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.288337 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.288587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.288559 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.288730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.288711 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2lqg\" (UniqueName: \"kubernetes.io/projected/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-kube-api-access-n2lqg\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.389535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.389496 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.389735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.389656 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.389805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.389786 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2lqg\" (UniqueName: \"kubernetes.io/projected/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-kube-api-access-n2lqg\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.390180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.390158 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.392013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.391995 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.442848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.442816 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2lqg\" (UniqueName: \"kubernetes.io/projected/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-kube-api-access-n2lqg\") pod \"maas-api-revoke-sub-3300fffd-qz9rr\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.539813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.539723 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:21.741143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.741094 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" Nov 02 20:08:21.741143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.741128 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr" event={"ID":"8943ff71-b8e8-437c-ab45-7bb85c9fd6a9","Type":"ContainerDied","Data":"6332c116ff2a153f25c774695bfd529894ca674dfea808fb393984a869ce5547"} Nov 02 20:08:21.741428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.741193 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6332c116ff2a153f25c774695bfd529894ca674dfea808fb393984a869ce5547" Nov 02 20:08:21.794014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.793918 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:08:21.795105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.795081 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:08:21.915489 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:21.915459 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr"] Nov 02 20:08:21.917121 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:21.917093 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f601a7_f7aa_4c7f_a0cc_029025f8db90.slice/crio-033c09edbbb9936edcf03c8cde3c6592efa81a8cbd909aa9fb02b9c3d5c471ee WatchSource:0}: Error finding container 033c09edbbb9936edcf03c8cde3c6592efa81a8cbd909aa9fb02b9c3d5c471ee: Status 404 returned error can't find the container with id 033c09edbbb9936edcf03c8cde3c6592efa81a8cbd909aa9fb02b9c3d5c471ee Nov 02 20:08:22.697827 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:08:22.697781 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:08:22.751563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:22.751529 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" event={"ID":"e9f601a7-f7aa-4c7f-a0cc-029025f8db90","Type":"ContainerStarted","Data":"675976b7900d2d466364e2f1f501e94e0f694ed032c883bcc85f7bd5bdbfe1b8"} Nov 02 20:08:22.751737 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:22.751592 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" event={"ID":"e9f601a7-f7aa-4c7f-a0cc-029025f8db90","Type":"ContainerStarted","Data":"033c09edbbb9936edcf03c8cde3c6592efa81a8cbd909aa9fb02b9c3d5c471ee"} Nov 02 20:08:22.858810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:22.858761 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" podStartSLOduration=1.8587413659999998 podStartE2EDuration="1.858741366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:22.856722355 +0000 UTC m=+1201.373494942" watchObservedRunningTime="2026-11-02 20:08:22.858741366 +0000 UTC m=+1201.375513955" Nov 02 20:08:23.760378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:23.760345 5800 generic.go:356] "Generic (PLEG): container finished" podID="e9f601a7-f7aa-4c7f-a0cc-029025f8db90" containerID="675976b7900d2d466364e2f1f501e94e0f694ed032c883bcc85f7bd5bdbfe1b8" exitCode=0 Nov 02 20:08:23.760802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:23.760440 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" event={"ID":"e9f601a7-f7aa-4c7f-a0cc-029025f8db90","Type":"ContainerDied","Data":"675976b7900d2d466364e2f1f501e94e0f694ed032c883bcc85f7bd5bdbfe1b8"} Nov 02 20:08:24.937868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:24.937841 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:25.037345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.037229 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-xks-ca\") pod \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " Nov 02 20:08:25.037526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.037352 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-openshift-ca\") pod \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " Nov 02 20:08:25.037526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.037449 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-kube-api-access-n2lqg\" (UniqueName: \"kubernetes.io/projected/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-kube-api-access-n2lqg\") pod \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\" (UID: \"e9f601a7-f7aa-4c7f-a0cc-029025f8db90\") " Nov 02 20:08:25.037742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.037715 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-openshift-ca" pod "e9f601a7-f7aa-4c7f-a0cc-029025f8db90" (UID: "e9f601a7-f7aa-4c7f-a0cc-029025f8db90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:25.038033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.038004 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:25.039564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.039541 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-kube-api-access-n2lqg" pod "e9f601a7-f7aa-4c7f-a0cc-029025f8db90" (UID: "e9f601a7-f7aa-4c7f-a0cc-029025f8db90"). InnerVolumeSpecName "kube-api-access-n2lqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:25.039664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.039580 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-xks-ca" pod "e9f601a7-f7aa-4c7f-a0cc-029025f8db90" (UID: "e9f601a7-f7aa-4c7f-a0cc-029025f8db90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:25.138952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.138913 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2lqg\" (UniqueName: \"kubernetes.io/projected/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-kube-api-access-n2lqg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:25.138952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.138943 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f601a7-f7aa-4c7f-a0cc-029025f8db90-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:25.777760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.777727 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" Nov 02 20:08:25.777908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.777758 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr" event={"ID":"e9f601a7-f7aa-4c7f-a0cc-029025f8db90","Type":"ContainerDied","Data":"033c09edbbb9936edcf03c8cde3c6592efa81a8cbd909aa9fb02b9c3d5c471ee"} Nov 02 20:08:25.777908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:25.777807 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033c09edbbb9936edcf03c8cde3c6592efa81a8cbd909aa9fb02b9c3d5c471ee" Nov 02 20:08:27.051674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:27.051638 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98"] Nov 02 20:08:27.097216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:27.097181 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e53c24-5kn98"] Nov 02 20:08:27.641397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:27.641356 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d871ed-04f2-4fff-bda8-016983c49e15" path="/var/lib/kubelet/pods/89d871ed-04f2-4fff-bda8-016983c49e15/volumes" Nov 02 20:08:31.965067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:31.965028 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc"] Nov 02 20:08:31.965607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:31.965592 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e9f601a7-f7aa-4c7f-a0cc-029025f8db90" containerName="revoke-keys" Nov 02 20:08:31.965652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:31.965608 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f601a7-f7aa-4c7f-a0cc-029025f8db90" containerName="revoke-keys" Nov 02 20:08:31.965748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:31.965739 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f601a7-f7aa-4c7f-a0cc-029025f8db90" containerName="revoke-keys" Nov 02 20:08:31.969816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:31.969785 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.022143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.022109 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc"] Nov 02 20:08:32.064476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.064418 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:08:32.064677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.064489 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:08:32.064677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.064534 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:08:32.065774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.065753 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:08:32.131093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.131061 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf827857-c1ef-445a-9001-f875c8751836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.131341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.131312 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zjq\" (UniqueName: \"kubernetes.io/projected/bf827857-c1ef-445a-9001-f875c8751836-kube-api-access-45zjq\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.131431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.131376 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf827857-c1ef-445a-9001-f875c8751836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.232651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.232555 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zjq\" (UniqueName: \"kubernetes.io/projected/bf827857-c1ef-445a-9001-f875c8751836-kube-api-access-45zjq\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.232651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.232621 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf827857-c1ef-445a-9001-f875c8751836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.232858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.232708 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf827857-c1ef-445a-9001-f875c8751836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.233535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.233503 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf827857-c1ef-445a-9001-f875c8751836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.235177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.235148 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf827857-c1ef-445a-9001-f875c8751836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.285822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.285791 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zjq\" (UniqueName: \"kubernetes.io/projected/bf827857-c1ef-445a-9001-f875c8751836-kube-api-access-45zjq\") pod \"maas-api-revoke-sub-d80ba10d-7j7nc\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.287615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.287589 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:32.655589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.655559 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc"] Nov 02 20:08:32.656639 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:32.656608 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf827857_c1ef_445a_9001_f875c8751836.slice/crio-1228e00d3dfe675589afb335f38c000103fee47323d92e18bdb41ac795625eca WatchSource:0}: Error finding container 1228e00d3dfe675589afb335f38c000103fee47323d92e18bdb41ac795625eca: Status 404 returned error can't find the container with id 1228e00d3dfe675589afb335f38c000103fee47323d92e18bdb41ac795625eca Nov 02 20:08:32.844391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:32.844355 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" event={"ID":"bf827857-c1ef-445a-9001-f875c8751836","Type":"ContainerStarted","Data":"1228e00d3dfe675589afb335f38c000103fee47323d92e18bdb41ac795625eca"} Nov 02 20:08:33.852188 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:33.852152 5800 generic.go:356] "Generic (PLEG): container finished" podID="bf827857-c1ef-445a-9001-f875c8751836" containerID="d0ff408af4fcb89a21c1b3ad8e7690cb95da3df9b046d4d4257b711025435280" exitCode=0 Nov 02 20:08:33.852648 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:33.852242 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" event={"ID":"bf827857-c1ef-445a-9001-f875c8751836","Type":"ContainerDied","Data":"d0ff408af4fcb89a21c1b3ad8e7690cb95da3df9b046d4d4257b711025435280"} Nov 02 20:08:35.016155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.016129 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:35.168041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.167949 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf827857-c1ef-445a-9001-f875c8751836-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf827857-c1ef-445a-9001-f875c8751836-maas-api-xks-ca\") pod \"bf827857-c1ef-445a-9001-f875c8751836\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " Nov 02 20:08:35.168242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.168059 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf827857-c1ef-445a-9001-f875c8751836-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf827857-c1ef-445a-9001-f875c8751836-maas-api-openshift-ca\") pod \"bf827857-c1ef-445a-9001-f875c8751836\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " Nov 02 20:08:35.168242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.168133 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf827857-c1ef-445a-9001-f875c8751836-kube-api-access-45zjq\" (UniqueName: \"kubernetes.io/projected/bf827857-c1ef-445a-9001-f875c8751836-kube-api-access-45zjq\") pod \"bf827857-c1ef-445a-9001-f875c8751836\" (UID: \"bf827857-c1ef-445a-9001-f875c8751836\") " Nov 02 20:08:35.168513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.168487 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf827857-c1ef-445a-9001-f875c8751836-maas-api-openshift-ca" pod "bf827857-c1ef-445a-9001-f875c8751836" (UID: "bf827857-c1ef-445a-9001-f875c8751836"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:35.168665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.168639 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf827857-c1ef-445a-9001-f875c8751836-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:35.170623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.170589 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf827857-c1ef-445a-9001-f875c8751836-maas-api-xks-ca" pod "bf827857-c1ef-445a-9001-f875c8751836" (UID: "bf827857-c1ef-445a-9001-f875c8751836"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:35.170754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.170720 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf827857-c1ef-445a-9001-f875c8751836-kube-api-access-45zjq" pod "bf827857-c1ef-445a-9001-f875c8751836" (UID: "bf827857-c1ef-445a-9001-f875c8751836"). InnerVolumeSpecName "kube-api-access-45zjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:35.187988 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.187959 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4"] Nov 02 20:08:35.188846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.188821 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bf827857-c1ef-445a-9001-f875c8751836" containerName="revoke-keys" Nov 02 20:08:35.188846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.188846 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf827857-c1ef-445a-9001-f875c8751836" containerName="revoke-keys" Nov 02 20:08:35.189049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.189036 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf827857-c1ef-445a-9001-f875c8751836" containerName="revoke-keys" Nov 02 20:08:35.193180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.193161 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.199434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.199416 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7"] Nov 02 20:08:35.203494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.203472 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.271700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.271664 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45zjq\" (UniqueName: \"kubernetes.io/projected/bf827857-c1ef-445a-9001-f875c8751836-kube-api-access-45zjq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:35.271932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.271910 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf827857-c1ef-445a-9001-f875c8751836-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:35.292361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.292328 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4"] Nov 02 20:08:35.295721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.295680 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7"] Nov 02 20:08:35.372675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.372623 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.372675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.372676 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.372940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.372756 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecbc55f-3527-4934-b18f-278ccdcd7515-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.372940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.372780 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69834227-930d-4f3b-af29-8e3d93c76e81-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.372940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.372810 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69834227-930d-4f3b-af29-8e3d93c76e81-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.373051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.372950 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.373051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373006 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.373155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373068 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.373155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373136 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.373325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373212 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752f2\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-kube-api-access-752f2\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.373325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373261 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.373402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373366 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.373443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373419 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54582\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-kube-api-access-54582\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.373483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373441 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.373483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373462 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.373582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373484 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.373582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373526 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.373708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.373636 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.474495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474404 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.474495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474440 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.474495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474456 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.474495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474476 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474510 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474534 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474561 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecbc55f-3527-4934-b18f-278ccdcd7515-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474587 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69834227-930d-4f3b-af29-8e3d93c76e81-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474628 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69834227-930d-4f3b-af29-8e3d93c76e81-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474674 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474709 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474772 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474803 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.474872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474863 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752f2\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-kube-api-access-752f2\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474923 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474959 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474980 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.474999 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475006 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-certs\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475092 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54582\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-kube-api-access-54582\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475130 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.475410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475315 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.475861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475515 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-credential-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.475861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475530 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69834227-930d-4f3b-af29-8e3d93c76e81-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.475861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475739 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-data\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.476020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.475987 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.476207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.476179 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecbc55f-3527-4934-b18f-278ccdcd7515-istiod-ca-cert\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.476306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.476235 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-socket\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.477900 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.477873 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.478035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.477998 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.478234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.478213 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-envoy\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.478714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.478685 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69834227-930d-4f3b-af29-8e3d93c76e81-istio-podinfo\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.528266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.528230 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.528683 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.528662 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54582\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-kube-api-access-54582\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.528763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.528680 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-istio-token\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.528763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.528720 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-752f2\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-kube-api-access-752f2\") pod \"e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.809178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.809137 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:35.817661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.817629 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:35.873618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.871643 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" event={"ID":"bf827857-c1ef-445a-9001-f875c8751836","Type":"ContainerDied","Data":"1228e00d3dfe675589afb335f38c000103fee47323d92e18bdb41ac795625eca"} Nov 02 20:08:35.873618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.871703 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1228e00d3dfe675589afb335f38c000103fee47323d92e18bdb41ac795625eca" Nov 02 20:08:35.873618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:35.871771 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc" Nov 02 20:08:36.229455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.229424 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4"] Nov 02 20:08:36.229826 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:36.229501 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecbc55f_3527_4934_b18f_278ccdcd7515.slice/crio-5bdd8b81aa5f37ad9ce1bc4c2f3914508acbadea204a3262c6f68ab2d4abb125 WatchSource:0}: Error finding container 5bdd8b81aa5f37ad9ce1bc4c2f3914508acbadea204a3262c6f68ab2d4abb125: Status 404 returned error can't find the container with id 5bdd8b81aa5f37ad9ce1bc4c2f3914508acbadea204a3262c6f68ab2d4abb125 Nov 02 20:08:36.232810 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:36.231892 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69834227_930d_4f3b_af29_8e3d93c76e81.slice/crio-dfe4330e1b463bafe7def424a92fc7c5f3b525755963d4e922ad20865d51a672 WatchSource:0}: Error finding container dfe4330e1b463bafe7def424a92fc7c5f3b525755963d4e922ad20865d51a672: Status 404 returned error can't find the container with id dfe4330e1b463bafe7def424a92fc7c5f3b525755963d4e922ad20865d51a672 Nov 02 20:08:36.232810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.232438 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:36.232810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.232496 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:36.232810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.232523 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:36.233428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.233406 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7"] Nov 02 20:08:36.234804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.234770 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:36.234881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.234842 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:36.235038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.234884 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:08:36.890307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.887593 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" event={"ID":"5ecbc55f-3527-4934-b18f-278ccdcd7515","Type":"ContainerStarted","Data":"3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1"} Nov 02 20:08:36.890307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.887659 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" event={"ID":"5ecbc55f-3527-4934-b18f-278ccdcd7515","Type":"ContainerStarted","Data":"5bdd8b81aa5f37ad9ce1bc4c2f3914508acbadea204a3262c6f68ab2d4abb125"} Nov 02 20:08:36.891950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.891455 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" event={"ID":"69834227-930d-4f3b-af29-8e3d93c76e81","Type":"ContainerStarted","Data":"173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2"} Nov 02 20:08:36.891950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:36.891515 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" event={"ID":"69834227-930d-4f3b-af29-8e3d93c76e81","Type":"ContainerStarted","Data":"dfe4330e1b463bafe7def424a92fc7c5f3b525755963d4e922ad20865d51a672"} Nov 02 20:08:37.001785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.001731 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podStartSLOduration=2.001713797 podStartE2EDuration="2.001713797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:36.99686679 +0000 UTC m=+1215.513639407" watchObservedRunningTime="2026-11-02 20:08:37.001713797 +0000 UTC m=+1215.518486383" Nov 02 20:08:37.112954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.112901 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podStartSLOduration=2.112887183 podStartE2EDuration="2.112887183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:37.108517057 +0000 UTC m=+1215.625289640" watchObservedRunningTime="2026-11-02 20:08:37.112887183 +0000 UTC m=+1215.629659767" Nov 02 20:08:37.194775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.194681 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z"] Nov 02 20:08:37.194936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.194882 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" containerID="cri-o://c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734" gracePeriod=30 Nov 02 20:08:37.809441 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.809404 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:37.811295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.811257 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:08:37.811404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.811346 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:08:37.818299 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.818255 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:37.819377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.819356 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:08:37.819456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:37.819402 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:08:38.810084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:38.810037 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:08:38.810595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:38.810105 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:08:38.818675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:38.818650 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:08:38.818861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:38.818834 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:08:39.809898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:39.809857 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:08:39.810084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:39.809917 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:08:39.818514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:39.818484 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:08:39.818865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:39.818554 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:08:40.813304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:40.811630 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:08:40.813304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:40.811691 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:08:40.819309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:40.818582 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:08:40.819309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:40.818638 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:08:41.810539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:41.810485 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:08:41.810701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:41.810569 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:08:41.817905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:41.817874 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:08:41.818048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:41.817920 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:08:42.583176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.583147 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:08:42.698865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.698833 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-token\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.698883 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istiod-ca-cert\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.698909 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-certs\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.698949 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-credential-socket\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.698971 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-envoy\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699024 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-socket\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699189 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-data\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699273 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-kube-api-access-sjrtb\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699335 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-podinfo\") pod \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\" (UID: \"4fbf9e3f-d603-4c75-ae01-f9c277be7676\") " Nov 02 20:08:42.699519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699462 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istiod-ca-cert" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:42.699570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699507 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-data" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:42.699630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699572 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-socket" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:42.699682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699667 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-certs" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:42.699978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.699949 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-credential-socket" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:42.700126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.700104 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.700196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.700134 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.700196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.700149 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.700196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.700163 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.700196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.700177 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.701682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.701652 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-envoy" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:08:42.702056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.702036 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-kube-api-access-sjrtb" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "kube-api-access-sjrtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:42.703065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.703045 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-token" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:42.703144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.703115 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-podinfo" pod "4fbf9e3f-d603-4c75-ae01-f9c277be7676" (UID: "4fbf9e3f-d603-4c75-ae01-f9c277be7676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:08:42.802010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.801907 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.802010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.801938 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.802010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.801949 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjrtb\" (UniqueName: \"kubernetes.io/projected/4fbf9e3f-d603-4c75-ae01-f9c277be7676-kube-api-access-sjrtb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.802010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.801958 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf9e3f-d603-4c75-ae01-f9c277be7676-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:42.810344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.810308 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 20:08:42.810481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.810378 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 20:08:42.818733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.818710 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 02 20:08:42.818850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.818766 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 02 20:08:42.985412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.985377 5800 generic.go:356] "Generic (PLEG): container finished" podID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerID="c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734" exitCode=0 Nov 02 20:08:42.985585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.985442 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" Nov 02 20:08:42.985585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.985467 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" event={"ID":"4fbf9e3f-d603-4c75-ae01-f9c277be7676","Type":"ContainerDied","Data":"c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734"} Nov 02 20:08:42.985585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.985534 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z" event={"ID":"4fbf9e3f-d603-4c75-ae01-f9c277be7676","Type":"ContainerDied","Data":"9f27199487eefb321ca64110dc983faf9d60835067cdd6713ca3027e2bd95853"} Nov 02 20:08:42.985585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.985548 5800 scope.go:122] "RemoveContainer" containerID="c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734" Nov 02 20:08:42.999727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:42.999707 5800 scope.go:122] "RemoveContainer" containerID="c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734" Nov 02 20:08:43.000082 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:08:43.000060 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734\": container with ID starting with c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734 not found: ID does not exist" containerID="c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734" Nov 02 20:08:43.000180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.000091 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734"} err="failed to get container status \"c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734\": rpc error: code = NotFound desc = could not find container \"c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734\": container with ID starting with c7837563240af87b6ee8dc0788373c579eb0e26d7c76f930515f357e7d04c734 not found: ID does not exist" Nov 02 20:08:43.144066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.143958 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z"] Nov 02 20:08:43.149548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.149519 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-flk9z"] Nov 02 20:08:43.627983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.627938 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" path="/var/lib/kubelet/pods/4fbf9e3f-d603-4c75-ae01-f9c277be7676/volumes" Nov 02 20:08:43.848061 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.848027 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:43.850054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.850032 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:43.856194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.856161 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:08:43.939861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.939782 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:43.940192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.940167 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:43.951325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:43.942702 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:08:45.043453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:45.043420 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss"] Nov 02 20:08:45.089718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:45.089686 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b149fe1-x89ss"] Nov 02 20:08:45.622132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:45.622098 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884c1be9-31f9-47ff-b41a-930e5e39b15d" path="/var/lib/kubelet/pods/884c1be9-31f9-47ff-b41a-930e5e39b15d/volumes" Nov 02 20:08:48.612767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.612728 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld"] Nov 02 20:08:48.613415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.613394 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" Nov 02 20:08:48.613524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.613414 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" Nov 02 20:08:48.613622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.613609 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbf9e3f-d603-4c75-ae01-f9c277be7676" containerName="istio-proxy" Nov 02 20:08:48.618793 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.618772 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.670141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.670106 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld"] Nov 02 20:08:48.712709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.712662 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:08:48.712709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.712678 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:08:48.712971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.712721 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:08:48.713942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.713911 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:08:48.762179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.762149 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.762179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.762193 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.762420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.762305 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqj6j\" (UniqueName: \"kubernetes.io/projected/9abd0b11-b69b-4eaf-ab81-2109705adf0a-kube-api-access-wqj6j\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.863043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.862961 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.863043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.863004 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.863043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.863031 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqj6j\" (UniqueName: \"kubernetes.io/projected/9abd0b11-b69b-4eaf-ab81-2109705adf0a-kube-api-access-wqj6j\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.863736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.863714 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.865390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.865364 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.916116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.916080 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqj6j\" (UniqueName: \"kubernetes.io/projected/9abd0b11-b69b-4eaf-ab81-2109705adf0a-kube-api-access-wqj6j\") pod \"maas-api-revoke-sub-401c97fd-xlqld\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:48.934382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:48.934351 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:49.310739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:49.310710 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld"] Nov 02 20:08:49.311785 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:49.311741 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abd0b11_b69b_4eaf_ab81_2109705adf0a.slice/crio-5b438b36715afb60f1b257d24471ea20e188076fa566b79f44e5defb68b42cd6 WatchSource:0}: Error finding container 5b438b36715afb60f1b257d24471ea20e188076fa566b79f44e5defb68b42cd6: Status 404 returned error can't find the container with id 5b438b36715afb60f1b257d24471ea20e188076fa566b79f44e5defb68b42cd6 Nov 02 20:08:50.111829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:50.111795 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" event={"ID":"9abd0b11-b69b-4eaf-ab81-2109705adf0a","Type":"ContainerStarted","Data":"7b86b6842f385c1c46310633314c367e04913b6d56e229c488141ed282e39a44"} Nov 02 20:08:50.112196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:50.111846 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" event={"ID":"9abd0b11-b69b-4eaf-ab81-2109705adf0a","Type":"ContainerStarted","Data":"5b438b36715afb60f1b257d24471ea20e188076fa566b79f44e5defb68b42cd6"} Nov 02 20:08:50.218646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:50.218595 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" podStartSLOduration=2.218581079 podStartE2EDuration="2.218581079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:50.217609069 +0000 UTC m=+1228.734381654" watchObservedRunningTime="2026-11-02 20:08:50.218581079 +0000 UTC m=+1228.735353664" Nov 02 20:08:51.119999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:51.119960 5800 generic.go:356] "Generic (PLEG): container finished" podID="9abd0b11-b69b-4eaf-ab81-2109705adf0a" containerID="7b86b6842f385c1c46310633314c367e04913b6d56e229c488141ed282e39a44" exitCode=0 Nov 02 20:08:51.120497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:51.120037 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" event={"ID":"9abd0b11-b69b-4eaf-ab81-2109705adf0a","Type":"ContainerDied","Data":"7b86b6842f385c1c46310633314c367e04913b6d56e229c488141ed282e39a44"} Nov 02 20:08:52.326305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.324312 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:52.405418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.405342 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abd0b11-b69b-4eaf-ab81-2109705adf0a-kube-api-access-wqj6j\" (UniqueName: \"kubernetes.io/projected/9abd0b11-b69b-4eaf-ab81-2109705adf0a-kube-api-access-wqj6j\") pod \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " Nov 02 20:08:52.405418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.405380 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-openshift-ca\") pod \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " Nov 02 20:08:52.405593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.405443 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-xks-ca\") pod \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\" (UID: \"9abd0b11-b69b-4eaf-ab81-2109705adf0a\") " Nov 02 20:08:52.405855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.405826 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-openshift-ca" pod "9abd0b11-b69b-4eaf-ab81-2109705adf0a" (UID: "9abd0b11-b69b-4eaf-ab81-2109705adf0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:08:52.407587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.407567 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abd0b11-b69b-4eaf-ab81-2109705adf0a-kube-api-access-wqj6j" pod "9abd0b11-b69b-4eaf-ab81-2109705adf0a" (UID: "9abd0b11-b69b-4eaf-ab81-2109705adf0a"). InnerVolumeSpecName "kube-api-access-wqj6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:08:52.407651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.407592 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-xks-ca" pod "9abd0b11-b69b-4eaf-ab81-2109705adf0a" (UID: "9abd0b11-b69b-4eaf-ab81-2109705adf0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:08:52.506855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.506820 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:52.506855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.506850 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqj6j\" (UniqueName: \"kubernetes.io/projected/9abd0b11-b69b-4eaf-ab81-2109705adf0a-kube-api-access-wqj6j\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:52.506855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:52.506861 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0b11-b69b-4eaf-ab81-2109705adf0a-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:08:52.698557 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:08:52.698469 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:08:53.144109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:53.144073 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" Nov 02 20:08:53.144109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:53.144097 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld" event={"ID":"9abd0b11-b69b-4eaf-ab81-2109705adf0a","Type":"ContainerDied","Data":"5b438b36715afb60f1b257d24471ea20e188076fa566b79f44e5defb68b42cd6"} Nov 02 20:08:53.144390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:53.144152 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b438b36715afb60f1b257d24471ea20e188076fa566b79f44e5defb68b42cd6" Nov 02 20:08:56.132958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.132920 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z"] Nov 02 20:08:56.133568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.133550 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9abd0b11-b69b-4eaf-ab81-2109705adf0a" containerName="revoke-keys" Nov 02 20:08:56.133568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.133568 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abd0b11-b69b-4eaf-ab81-2109705adf0a" containerName="revoke-keys" Nov 02 20:08:56.133702 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.133691 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abd0b11-b69b-4eaf-ab81-2109705adf0a" containerName="revoke-keys" Nov 02 20:08:56.137506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.137484 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.235360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.235328 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-bbee67\"/\"distinct-w1-bbee67-kserve-self-signed-certs\"" Nov 02 20:08:56.241342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.241313 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z"] Nov 02 20:08:56.253113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.253077 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-model-cache\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.253300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.253132 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-dshm\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.253413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.253394 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hprmk\" (UniqueName: \"kubernetes.io/projected/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-kube-api-access-hprmk\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.253475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.253427 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tls-certs\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.253475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.253456 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-home\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.253575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.253554 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tmp-dir\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.355382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355344 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hprmk\" (UniqueName: \"kubernetes.io/projected/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-kube-api-access-hprmk\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.355382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355381 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tls-certs\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.355635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355400 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-home\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.355635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355438 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tmp-dir\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.355635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355517 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-model-cache\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.355798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355726 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-dshm\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.355953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355930 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-home\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.356011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.355949 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-model-cache\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.356229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.356194 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tmp-dir\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.357858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.357826 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-dshm\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.357987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.357955 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tls-certs\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.408842 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.408765 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hprmk\" (UniqueName: \"kubernetes.io/projected/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-kube-api-access-hprmk\") pod \"distinct-w1-bbee67-kserve-8485c65785-4d44z\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.450648 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.450612 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:56.737910 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.737835 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb"] Nov 02 20:08:56.744472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.744447 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.792415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.792386 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb"] Nov 02 20:08:56.792842 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.792819 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:08:56.792958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.792944 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:08:56.792998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.792970 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:08:56.792998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.792980 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:08:56.847626 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:08:56.847598 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75ed7183_0be5_4c22_8795_a4bfedd3d5cd.slice/crio-f8b44eaf59f92fafa404bedda0bd1b51052ff95c997cd6f69a30ebda0dd106f6 WatchSource:0}: Error finding container f8b44eaf59f92fafa404bedda0bd1b51052ff95c997cd6f69a30ebda0dd106f6: Status 404 returned error can't find the container with id f8b44eaf59f92fafa404bedda0bd1b51052ff95c997cd6f69a30ebda0dd106f6 Nov 02 20:08:56.848244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.848220 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z"] Nov 02 20:08:56.870795 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.870767 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hc6\" (UniqueName: \"kubernetes.io/projected/57a7c678-6bd4-485d-960d-fde51c690d14-kube-api-access-t4hc6\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.870947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.870921 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.871093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.871071 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.972439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.972402 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hc6\" (UniqueName: \"kubernetes.io/projected/57a7c678-6bd4-485d-960d-fde51c690d14-kube-api-access-t4hc6\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.972648 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.972475 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.972648 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.972546 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.973381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.973348 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:56.975002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:56.974980 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:57.025759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:57.025726 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hc6\" (UniqueName: \"kubernetes.io/projected/57a7c678-6bd4-485d-960d-fde51c690d14-kube-api-access-t4hc6\") pod \"maas-api-revoke-sub-4c9db841-dclnb\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:57.068244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:57.068209 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:08:57.185731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:57.185687 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" event={"ID":"75ed7183-0be5-4c22-8795-a4bfedd3d5cd","Type":"ContainerStarted","Data":"316f12a92a6201495f2160149d8b260d4de665a8ee5c4c5d1962d9f048a5a405"} Nov 02 20:08:57.186212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:57.185742 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" event={"ID":"75ed7183-0be5-4c22-8795-a4bfedd3d5cd","Type":"ContainerStarted","Data":"f8b44eaf59f92fafa404bedda0bd1b51052ff95c997cd6f69a30ebda0dd106f6"} Nov 02 20:08:57.294666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:57.294556 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podStartSLOduration=1.294542256 podStartE2EDuration="1.294542256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:57.292011056 +0000 UTC m=+1235.808783636" watchObservedRunningTime="2026-11-02 20:08:57.294542256 +0000 UTC m=+1235.811314865" Nov 02 20:08:57.450063 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:57.444557 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb"] Nov 02 20:08:57.456070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:57.456035 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:08:58.194875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:58.194832 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" event={"ID":"57a7c678-6bd4-485d-960d-fde51c690d14","Type":"ContainerStarted","Data":"4ebc443e112470a7b20c470fb28a3a48ccca53311576013eb886e7fe333f60d2"} Nov 02 20:08:58.195305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:58.194897 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" event={"ID":"57a7c678-6bd4-485d-960d-fde51c690d14","Type":"ContainerStarted","Data":"041c16c1eaea41b4e80fdeb182a665eafcde45a2c1489b682bb37ba1a139291a"} Nov 02 20:08:58.303313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:58.303247 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" podStartSLOduration=2.303232283 podStartE2EDuration="2.303232283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:08:58.30038623 +0000 UTC m=+1236.817158820" watchObservedRunningTime="2026-11-02 20:08:58.303232283 +0000 UTC m=+1236.820004867" Nov 02 20:08:59.205898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:59.205859 5800 generic.go:356] "Generic (PLEG): container finished" podID="57a7c678-6bd4-485d-960d-fde51c690d14" containerID="4ebc443e112470a7b20c470fb28a3a48ccca53311576013eb886e7fe333f60d2" exitCode=0 Nov 02 20:08:59.206591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:08:59.205934 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" event={"ID":"57a7c678-6bd4-485d-960d-fde51c690d14","Type":"ContainerDied","Data":"4ebc443e112470a7b20c470fb28a3a48ccca53311576013eb886e7fe333f60d2"} Nov 02 20:09:00.818238 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:00.818211 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:09:00.964926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:00.964828 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a7c678-6bd4-485d-960d-fde51c690d14-kube-api-access-t4hc6\" (UniqueName: \"kubernetes.io/projected/57a7c678-6bd4-485d-960d-fde51c690d14-kube-api-access-t4hc6\") pod \"57a7c678-6bd4-485d-960d-fde51c690d14\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " Nov 02 20:09:00.965089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:00.964968 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-xks-ca\") pod \"57a7c678-6bd4-485d-960d-fde51c690d14\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " Nov 02 20:09:00.965089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:00.965014 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-openshift-ca\") pod \"57a7c678-6bd4-485d-960d-fde51c690d14\" (UID: \"57a7c678-6bd4-485d-960d-fde51c690d14\") " Nov 02 20:09:00.965683 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:00.965651 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-openshift-ca" pod "57a7c678-6bd4-485d-960d-fde51c690d14" (UID: "57a7c678-6bd4-485d-960d-fde51c690d14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:00.967330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:00.967271 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a7c678-6bd4-485d-960d-fde51c690d14-kube-api-access-t4hc6" pod "57a7c678-6bd4-485d-960d-fde51c690d14" (UID: "57a7c678-6bd4-485d-960d-fde51c690d14"). InnerVolumeSpecName "kube-api-access-t4hc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:00.968141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:00.968109 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-xks-ca" pod "57a7c678-6bd4-485d-960d-fde51c690d14" (UID: "57a7c678-6bd4-485d-960d-fde51c690d14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:01.066771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:01.066730 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:01.066771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:01.066765 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hc6\" (UniqueName: \"kubernetes.io/projected/57a7c678-6bd4-485d-960d-fde51c690d14-kube-api-access-t4hc6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:01.066771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:01.066776 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a7c678-6bd4-485d-960d-fde51c690d14-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:01.256837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:01.256803 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" Nov 02 20:09:01.257007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:01.256837 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb" event={"ID":"57a7c678-6bd4-485d-960d-fde51c690d14","Type":"ContainerDied","Data":"041c16c1eaea41b4e80fdeb182a665eafcde45a2c1489b682bb37ba1a139291a"} Nov 02 20:09:01.257007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:01.256884 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="041c16c1eaea41b4e80fdeb182a665eafcde45a2c1489b682bb37ba1a139291a" Nov 02 20:09:05.541507 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.541466 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg"] Nov 02 20:09:05.542380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.542360 5800 cpu_manager.go:414] "containerMap: removing container" podUID="57a7c678-6bd4-485d-960d-fde51c690d14" containerName="revoke-keys" Nov 02 20:09:05.542380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.542379 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a7c678-6bd4-485d-960d-fde51c690d14" containerName="revoke-keys" Nov 02 20:09:05.542539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.542481 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a7c678-6bd4-485d-960d-fde51c690d14" containerName="revoke-keys" Nov 02 20:09:05.546373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.546352 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.629075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.629039 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4"] Nov 02 20:09:05.640079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.640048 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.652822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.652792 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg"] Nov 02 20:09:05.686410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.686377 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4"] Nov 02 20:09:05.688828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.688804 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:09:05.688989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.688798 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:05.689072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.689052 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:09:05.689684 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.689664 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:09:05.729505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729477 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.729673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729522 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.729673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729598 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.729673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729636 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200795e0-806f-4df7-98d2-a3594072c700-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.729820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729723 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.729820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729751 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.729820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729770 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.729929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729911 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200795e0-806f-4df7-98d2-a3594072c700-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.729991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729975 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.730032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.729999 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.730032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.730022 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrt2g\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-kube-api-access-nrt2g\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.730126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.730047 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmqf\" (UniqueName: \"kubernetes.io/projected/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-kube-api-access-4kmqf\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.831589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831493 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmqf\" (UniqueName: \"kubernetes.io/projected/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-kube-api-access-4kmqf\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.831771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831596 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.831771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831627 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.831771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831646 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.831771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831669 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200795e0-806f-4df7-98d2-a3594072c700-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.831771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831703 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.831771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831741 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.831771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831765 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831864 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200795e0-806f-4df7-98d2-a3594072c700-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831928 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831951 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.831989 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrt2g\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-kube-api-access-nrt2g\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.832195 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.832463 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.832580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.832541 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832645 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.832618 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200795e0-806f-4df7-98d2-a3594072c700-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.832786 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.832884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.832846 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.834316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.834276 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.834629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.834605 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.835029 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.835008 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200795e0-806f-4df7-98d2-a3594072c700-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.885052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.885018 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.885310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.885270 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmqf\" (UniqueName: \"kubernetes.io/projected/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-kube-api-access-4kmqf\") pod \"maas-api-revoke-sub-b3e7d177-6jmh4\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:05.885361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.885270 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrt2g\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-kube-api-access-nrt2g\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:05.960229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:05.960197 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:06.161441 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.161342 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:06.342082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.342041 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4"] Nov 02 20:09:06.342511 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:06.342483 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61d2a0a_440b_43ad_b43c_f5fc4171c7aa.slice/crio-c49721f5d3c445ddc02c0934b5603fe7c7ef11a6501dee58eba4f4b80a27be2d WatchSource:0}: Error finding container c49721f5d3c445ddc02c0934b5603fe7c7ef11a6501dee58eba4f4b80a27be2d: Status 404 returned error can't find the container with id c49721f5d3c445ddc02c0934b5603fe7c7ef11a6501dee58eba4f4b80a27be2d Nov 02 20:09:06.451416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.451322 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:09:06.465268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.465237 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:09:06.533980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.533945 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg"] Nov 02 20:09:06.540539 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:06.540510 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200795e0_806f_4df7_98d2_a3594072c700.slice/crio-29317abbc40f36d212a0fccaf6dfe7f2b22257e88701d33a770d06c152879c4d WatchSource:0}: Error finding container 29317abbc40f36d212a0fccaf6dfe7f2b22257e88701d33a770d06c152879c4d: Status 404 returned error can't find the container with id 29317abbc40f36d212a0fccaf6dfe7f2b22257e88701d33a770d06c152879c4d Nov 02 20:09:06.543490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.543453 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:06.544117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.543539 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:06.544117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:06.543580 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:07.303641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:07.303606 5800 generic.go:356] "Generic (PLEG): container finished" podID="c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" containerID="db8b4f9ec1551cee712153b3edc08a760a1fab54d64e7319b2b8b7c609e443b8" exitCode=0 Nov 02 20:09:07.303839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:07.303700 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" event={"ID":"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa","Type":"ContainerDied","Data":"db8b4f9ec1551cee712153b3edc08a760a1fab54d64e7319b2b8b7c609e443b8"} Nov 02 20:09:07.303839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:07.303793 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" event={"ID":"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa","Type":"ContainerStarted","Data":"c49721f5d3c445ddc02c0934b5603fe7c7ef11a6501dee58eba4f4b80a27be2d"} Nov 02 20:09:07.306066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:07.306024 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" event={"ID":"200795e0-806f-4df7-98d2-a3594072c700","Type":"ContainerStarted","Data":"bdf2786840974519087e29e6ae2419a3cabd81945085548902d6439a2a5c1214"} Nov 02 20:09:07.306223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:07.306107 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" event={"ID":"200795e0-806f-4df7-98d2-a3594072c700","Type":"ContainerStarted","Data":"29317abbc40f36d212a0fccaf6dfe7f2b22257e88701d33a770d06c152879c4d"} Nov 02 20:09:07.325307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:07.324021 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:09:07.523851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:07.523800 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podStartSLOduration=2.523787197 podStartE2EDuration="2.523787197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:07.523528991 +0000 UTC m=+1246.040301578" watchObservedRunningTime="2026-11-02 20:09:07.523787197 +0000 UTC m=+1246.040559781" Nov 02 20:09:08.161930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.161893 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:08.163589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.163558 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:08.163720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.163624 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:08.314952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.314865 5800 generic.go:356] "Generic (PLEG): container finished" podID="b4e9df5c-b0ff-456e-b298-d2115c99711d" containerID="0f51a9bbcb206f991f802964adf82faaffe1cf4fbd9b394555e2e5934fa95cd1" exitCode=0 Nov 02 20:09:08.314952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.314935 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" event={"ID":"b4e9df5c-b0ff-456e-b298-d2115c99711d","Type":"ContainerDied","Data":"0f51a9bbcb206f991f802964adf82faaffe1cf4fbd9b394555e2e5934fa95cd1"} Nov 02 20:09:08.580052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.579768 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:08.588684 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.588662 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-xks-ca\") pod \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " Nov 02 20:09:08.589199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.588969 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-kube-api-access-4kmqf\" (UniqueName: \"kubernetes.io/projected/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-kube-api-access-4kmqf\") pod \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " Nov 02 20:09:08.589199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.589040 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-openshift-ca\") pod \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\" (UID: \"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa\") " Nov 02 20:09:08.589807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.589782 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-openshift-ca" pod "c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" (UID: "c61d2a0a-440b-43ad-b43c-f5fc4171c7aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:08.590893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.590856 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-xks-ca" pod "c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" (UID: "c61d2a0a-440b-43ad-b43c-f5fc4171c7aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:08.590999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.590976 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-kube-api-access-4kmqf" pod "c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" (UID: "c61d2a0a-440b-43ad-b43c-f5fc4171c7aa"). InnerVolumeSpecName "kube-api-access-4kmqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:08.690930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.690893 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kmqf\" (UniqueName: \"kubernetes.io/projected/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-kube-api-access-4kmqf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:08.691162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.691136 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:08.691162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:08.691165 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:09.059079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.059017 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb"] Nov 02 20:09:09.061637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.061605 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125df16a-pg7mb"] Nov 02 20:09:09.163083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.162737 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:09.163083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.162813 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:09.323827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.323738 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" Nov 02 20:09:09.323827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.323758 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4" event={"ID":"c61d2a0a-440b-43ad-b43c-f5fc4171c7aa","Type":"ContainerDied","Data":"c49721f5d3c445ddc02c0934b5603fe7c7ef11a6501dee58eba4f4b80a27be2d"} Nov 02 20:09:09.323827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.323807 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c49721f5d3c445ddc02c0934b5603fe7c7ef11a6501dee58eba4f4b80a27be2d" Nov 02 20:09:09.570858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.570544 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:09:09.603410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.603340 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-trusted-ca-bundle\") pod \"b4e9df5c-b0ff-456e-b298-d2115c99711d\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " Nov 02 20:09:09.603538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.603460 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e9df5c-b0ff-456e-b298-d2115c99711d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4e9df5c-b0ff-456e-b298-d2115c99711d-archives-path\") pod \"b4e9df5c-b0ff-456e-b298-d2115c99711d\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " Nov 02 20:09:09.603538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.603486 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e9df5c-b0ff-456e-b298-d2115c99711d-kube-api-access-h8m2b\" (UniqueName: \"kubernetes.io/projected/b4e9df5c-b0ff-456e-b298-d2115c99711d-kube-api-access-h8m2b\") pod \"b4e9df5c-b0ff-456e-b298-d2115c99711d\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " Nov 02 20:09:09.603538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.603523 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-service-ca-bundle\") pod \"b4e9df5c-b0ff-456e-b298-d2115c99711d\" (UID: \"b4e9df5c-b0ff-456e-b298-d2115c99711d\") " Nov 02 20:09:09.603737 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.603710 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-trusted-ca-bundle" pod "b4e9df5c-b0ff-456e-b298-d2115c99711d" (UID: "b4e9df5c-b0ff-456e-b298-d2115c99711d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:09.603893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.603881 5800 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-trusted-ca-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:09.604047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.604031 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e9df5c-b0ff-456e-b298-d2115c99711d-archives-path" pod "b4e9df5c-b0ff-456e-b298-d2115c99711d" (UID: "b4e9df5c-b0ff-456e-b298-d2115c99711d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:09:09.604217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.604203 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-service-ca-bundle" pod "b4e9df5c-b0ff-456e-b298-d2115c99711d" (UID: "b4e9df5c-b0ff-456e-b298-d2115c99711d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:09.605957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.605933 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e9df5c-b0ff-456e-b298-d2115c99711d-kube-api-access-h8m2b" pod "b4e9df5c-b0ff-456e-b298-d2115c99711d" (UID: "b4e9df5c-b0ff-456e-b298-d2115c99711d"). InnerVolumeSpecName "kube-api-access-h8m2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:09.623252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.623227 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84528565-5aba-4954-af8d-4c77a3307c8d" path="/var/lib/kubelet/pods/84528565-5aba-4954-af8d-4c77a3307c8d/volumes" Nov 02 20:09:09.705881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.705850 5800 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b4e9df5c-b0ff-456e-b298-d2115c99711d-archives-path\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:09.706057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.705885 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8m2b\" (UniqueName: \"kubernetes.io/projected/b4e9df5c-b0ff-456e-b298-d2115c99711d-kube-api-access-h8m2b\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:09.706057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:09.705903 5800 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4e9df5c-b0ff-456e-b298-d2115c99711d-service-ca-bundle\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:10.162508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:10.162474 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:10.162707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:10.162529 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:10.369429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:10.369383 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" event={"ID":"b4e9df5c-b0ff-456e-b298-d2115c99711d","Type":"ContainerDied","Data":"c9d73aa1693c26302048aa61af725148dba052a1d3c5120737eb2576fb5880ae"} Nov 02 20:09:10.369904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:10.369481 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d73aa1693c26302048aa61af725148dba052a1d3c5120737eb2576fb5880ae" Nov 02 20:09:10.369904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:10.369564 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xx7zt-m4wgl" Nov 02 20:09:11.162269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:11.162236 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:11.162449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:11.162323 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:12.186305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:12.180056 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:12.186305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:12.180129 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:12.563790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:12.563739 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:12.626417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:12.626365 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:12.784835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:12.784788 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:12.806447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:12.806397 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 20:09:12.935510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:12.935393 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:13.172306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.168713 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:13.172306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.168780 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:13.326305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.325424 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:13.370634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.370583 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:13.447307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.428678 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 20:09:13.573306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.572870 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 20:09:13.577354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.573554 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:09:13.583319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.580401 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:13.583319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.580468 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:13.631300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.627719 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:13.669307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.663506 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 20:09:13.908365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.907696 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 20:09:13.912322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.909326 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:09:14.017318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:13.990322 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 20:09:14.110298 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.110254 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:09:14.205363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.166625 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:14.205665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.205628 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:14.205823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.205806 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": context deadline exceeded" Nov 02 20:09:14.372307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.371391 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:14.411361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.382755 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:09:14.411361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.409649 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:14.414426 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.414394 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:09:14.435333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.431540 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:14.435333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.431889 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:09:14.596557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.593804 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:14.966583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:14.966458 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:15.050307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.006166 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:15.098753 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.040351 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 02 20:09:15.099004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.054300 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:15.099122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.061514 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:15.325325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.325130 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:15.408544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.408511 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:09:15.411259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.411230 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Nov 02 20:09:15.480476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.480437 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Nov 02 20:09:15.480735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.469022 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:09:15.480920 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.480905 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:09:15.494305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.440417 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:15.514581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.473900 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:15.619446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.618706 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:09:15.619628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.619512 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 20:09:15.667314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.666599 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:09:15.671354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:15.669836 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:09:16.294083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.294049 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:16.295366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.295337 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:16.296241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.296217 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:09:16.410000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.409951 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq"] Nov 02 20:09:16.410595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.410577 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b4e9df5c-b0ff-456e-b298-d2115c99711d" containerName="insights-gathering" Nov 02 20:09:16.410595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.410594 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e9df5c-b0ff-456e-b298-d2115c99711d" containerName="insights-gathering" Nov 02 20:09:16.410738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.410638 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" containerName="revoke-keys" Nov 02 20:09:16.410738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.410646 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" containerName="revoke-keys" Nov 02 20:09:16.410844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.410742 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" containerName="revoke-keys" Nov 02 20:09:16.410844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.410753 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e9df5c-b0ff-456e-b298-d2115c99711d" containerName="insights-gathering" Nov 02 20:09:16.415458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.415436 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.467439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.467405 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq"] Nov 02 20:09:16.467650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.467636 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:09:16.467732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.467706 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:09:16.467788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.467773 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:09:16.467847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.467800 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:16.479723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.479691 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.479904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.479838 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.479965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.479907 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdwp2\" (UniqueName: \"kubernetes.io/projected/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-kube-api-access-rdwp2\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.580826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.580725 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.580992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.580933 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.581060 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.581021 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdwp2\" (UniqueName: \"kubernetes.io/projected/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-kube-api-access-rdwp2\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.581592 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.581565 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.583444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.583423 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.639455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.639421 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdwp2\" (UniqueName: \"kubernetes.io/projected/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-kube-api-access-rdwp2\") pod \"maas-api-revoke-sub-39157f67-vt5qq\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:16.729704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:16.729664 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:17.121994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:17.121955 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq"] Nov 02 20:09:17.122150 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:17.122097 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5005ac4_f3d0_4e2d_80fb_585c6fee08bb.slice/crio-640388b1f9ea5bd4647d2dcb0dca44d8c83f86407a6e63f575ed6c42f4ad3c05 WatchSource:0}: Error finding container 640388b1f9ea5bd4647d2dcb0dca44d8c83f86407a6e63f575ed6c42f4ad3c05: Status 404 returned error can't find the container with id 640388b1f9ea5bd4647d2dcb0dca44d8c83f86407a6e63f575ed6c42f4ad3c05 Nov 02 20:09:17.307260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:17.307223 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" event={"ID":"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb","Type":"ContainerStarted","Data":"640388b1f9ea5bd4647d2dcb0dca44d8c83f86407a6e63f575ed6c42f4ad3c05"} Nov 02 20:09:18.315479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:18.315434 5800 generic.go:356] "Generic (PLEG): container finished" podID="c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" containerID="e8776c9176940c3b43857a78c52d72a2f05e6d981e0f28b14e85175088491d2e" exitCode=0 Nov 02 20:09:18.315892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:18.315519 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" event={"ID":"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb","Type":"ContainerDied","Data":"e8776c9176940c3b43857a78c52d72a2f05e6d981e0f28b14e85175088491d2e"} Nov 02 20:09:19.488201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.488166 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:19.516809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.516780 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-xks-ca\") pod \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " Nov 02 20:09:19.516983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.516890 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-kube-api-access-rdwp2\" (UniqueName: \"kubernetes.io/projected/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-kube-api-access-rdwp2\") pod \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " Nov 02 20:09:19.517077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.517002 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-openshift-ca\") pod \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\" (UID: \"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb\") " Nov 02 20:09:19.517431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.517403 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-openshift-ca" pod "c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" (UID: "c5005ac4-f3d0-4e2d-80fb-585c6fee08bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:19.518940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.518914 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-xks-ca" pod "c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" (UID: "c5005ac4-f3d0-4e2d-80fb-585c6fee08bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:19.519021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.518974 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-kube-api-access-rdwp2" pod "c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" (UID: "c5005ac4-f3d0-4e2d-80fb-585c6fee08bb"). InnerVolumeSpecName "kube-api-access-rdwp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:19.619805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.618841 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdwp2\" (UniqueName: \"kubernetes.io/projected/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-kube-api-access-rdwp2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:19.619805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.618881 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:19.619805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:19.618899 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:20.022459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.022425 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht"] Nov 02 20:09:20.022992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.022969 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" containerName="revoke-keys" Nov 02 20:09:20.022992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.022983 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" containerName="revoke-keys" Nov 02 20:09:20.023187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.023112 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" containerName="revoke-keys" Nov 02 20:09:20.026667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.026647 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.077348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.077307 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht"] Nov 02 20:09:20.126065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.126022 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.126234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.126092 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.126234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.126129 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pc6\" (UniqueName: \"kubernetes.io/projected/3877ccbe-0724-4170-bb40-76a54c5affc4-kube-api-access-v5pc6\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.226939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.226902 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.227127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.226948 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.227127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.226975 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pc6\" (UniqueName: \"kubernetes.io/projected/3877ccbe-0724-4170-bb40-76a54c5affc4-kube-api-access-v5pc6\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.227716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.227689 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.229215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.229177 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.280630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.280563 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pc6\" (UniqueName: \"kubernetes.io/projected/3877ccbe-0724-4170-bb40-76a54c5affc4-kube-api-access-v5pc6\") pod \"maas-api-revoke-sub-a098493b-c4wht\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.332857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.332825 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" Nov 02 20:09:20.333045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.332856 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq" event={"ID":"c5005ac4-f3d0-4e2d-80fb-585c6fee08bb","Type":"ContainerDied","Data":"640388b1f9ea5bd4647d2dcb0dca44d8c83f86407a6e63f575ed6c42f4ad3c05"} Nov 02 20:09:20.333045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.332905 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640388b1f9ea5bd4647d2dcb0dca44d8c83f86407a6e63f575ed6c42f4ad3c05" Nov 02 20:09:20.341502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.341476 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:20.670483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.670388 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh"] Nov 02 20:09:20.676067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.676043 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.718877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.718844 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht"] Nov 02 20:09:20.732380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732352 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732406 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732528 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732565 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732596 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732737 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8h9z\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-kube-api-access-t8h9z\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732778 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732813 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.732920 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.732883 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.773145 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.772635 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh"] Nov 02 20:09:20.833819 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.833783 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.833841 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.833865 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.833888 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.833917 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834005 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8h9z\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-kube-api-access-t8h9z\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834038 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834075 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834159 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834335 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834402 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834608 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834580 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834708 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.834819 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.834799 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.836239 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.836219 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.836519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.836502 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.887009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.886978 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.887141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.887125 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8h9z\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-kube-api-access-t8h9z\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:20.993035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:20.993002 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:21.342765 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:21.342726 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" event={"ID":"3877ccbe-0724-4170-bb40-76a54c5affc4","Type":"ContainerStarted","Data":"4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649"} Nov 02 20:09:21.342967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:21.342789 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" event={"ID":"3877ccbe-0724-4170-bb40-76a54c5affc4","Type":"ContainerStarted","Data":"51e1667507d70185eccf83c20c93ba58620107e981d3fedb4cbc2d7e54fc0be0"} Nov 02 20:09:21.415731 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:21.415697 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded74e3bc_2332_47fe_9276_6e4aa8161f14.slice/crio-eab75f6dda9e85a2796acdf77ecb8f65cf5c53e5ede2f5526d598855acc333b1 WatchSource:0}: Error finding container eab75f6dda9e85a2796acdf77ecb8f65cf5c53e5ede2f5526d598855acc333b1: Status 404 returned error can't find the container with id eab75f6dda9e85a2796acdf77ecb8f65cf5c53e5ede2f5526d598855acc333b1 Nov 02 20:09:21.416131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:21.416113 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh"] Nov 02 20:09:21.418161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:21.418127 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:21.418245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:21.418191 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:21.418245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:21.418222 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:21.450922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:21.450871 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" podStartSLOduration=2.450856146 podStartE2EDuration="2.450856146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:21.448120463 +0000 UTC m=+1259.964893047" watchObservedRunningTime="2026-11-02 20:09:21.450856146 +0000 UTC m=+1259.967628964" Nov 02 20:09:22.360346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.360235 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" event={"ID":"ed74e3bc-2332-47fe-9276-6e4aa8161f14","Type":"ContainerStarted","Data":"1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f"} Nov 02 20:09:22.360346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.360322 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" event={"ID":"ed74e3bc-2332-47fe-9276-6e4aa8161f14","Type":"ContainerStarted","Data":"eab75f6dda9e85a2796acdf77ecb8f65cf5c53e5ede2f5526d598855acc333b1"} Nov 02 20:09:22.362558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.362530 5800 generic.go:356] "Generic (PLEG): container finished" podID="3877ccbe-0724-4170-bb40-76a54c5affc4" containerID="4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649" exitCode=0 Nov 02 20:09:22.362669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.362610 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" event={"ID":"3877ccbe-0724-4170-bb40-76a54c5affc4","Type":"ContainerDied","Data":"4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649"} Nov 02 20:09:22.474320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.474243 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podStartSLOduration=2.474225382 podStartE2EDuration="2.474225382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:22.470231455 +0000 UTC m=+1260.987004039" watchObservedRunningTime="2026-11-02 20:09:22.474225382 +0000 UTC m=+1260.990997974" Nov 02 20:09:22.696906 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:22.696817 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:09:22.993874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.993780 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:22.995810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.995779 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:09:22.995949 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:22.995832 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 20:09:23.596969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.596939 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:23.673914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.673828 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-openshift-ca\") pod \"3877ccbe-0724-4170-bb40-76a54c5affc4\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " Nov 02 20:09:23.674132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.674113 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3877ccbe-0724-4170-bb40-76a54c5affc4-kube-api-access-v5pc6\" (UniqueName: \"kubernetes.io/projected/3877ccbe-0724-4170-bb40-76a54c5affc4-kube-api-access-v5pc6\") pod \"3877ccbe-0724-4170-bb40-76a54c5affc4\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " Nov 02 20:09:23.674509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.674490 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-xks-ca\") pod \"3877ccbe-0724-4170-bb40-76a54c5affc4\" (UID: \"3877ccbe-0724-4170-bb40-76a54c5affc4\") " Nov 02 20:09:23.674726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.674696 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-openshift-ca" pod "3877ccbe-0724-4170-bb40-76a54c5affc4" (UID: "3877ccbe-0724-4170-bb40-76a54c5affc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:23.675271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.675251 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:23.677946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.676088 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3877ccbe-0724-4170-bb40-76a54c5affc4-kube-api-access-v5pc6" pod "3877ccbe-0724-4170-bb40-76a54c5affc4" (UID: "3877ccbe-0724-4170-bb40-76a54c5affc4"). InnerVolumeSpecName "kube-api-access-v5pc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:23.680551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.678736 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-xks-ca" pod "3877ccbe-0724-4170-bb40-76a54c5affc4" (UID: "3877ccbe-0724-4170-bb40-76a54c5affc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:23.778133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.778088 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5pc6\" (UniqueName: \"kubernetes.io/projected/3877ccbe-0724-4170-bb40-76a54c5affc4-kube-api-access-v5pc6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:23.778133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.778132 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3877ccbe-0724-4170-bb40-76a54c5affc4-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:23.994231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.994137 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:09:23.994231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:23.994213 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 20:09:24.056378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:24.056333 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj"] Nov 02 20:09:24.061010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:24.060978 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb0715a-jjzjj"] Nov 02 20:09:24.389198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:24.389162 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" Nov 02 20:09:24.389198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:24.389183 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht" event={"ID":"3877ccbe-0724-4170-bb40-76a54c5affc4","Type":"ContainerDied","Data":"51e1667507d70185eccf83c20c93ba58620107e981d3fedb4cbc2d7e54fc0be0"} Nov 02 20:09:24.389470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:24.389238 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e1667507d70185eccf83c20c93ba58620107e981d3fedb4cbc2d7e54fc0be0" Nov 02 20:09:24.993672 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:24.993632 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:09:24.994082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:24.993706 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 20:09:25.623833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.623796 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f290f2f3-bef2-41d7-a1ea-65b9584bfc09" path="/var/lib/kubelet/pods/f290f2f3-bef2-41d7-a1ea-65b9584bfc09/volumes" Nov 02 20:09:25.671501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.671464 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps"] Nov 02 20:09:25.672083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.672062 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3877ccbe-0724-4170-bb40-76a54c5affc4" containerName="revoke-keys" Nov 02 20:09:25.672083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.672078 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3877ccbe-0724-4170-bb40-76a54c5affc4" containerName="revoke-keys" Nov 02 20:09:25.672231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.672195 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3877ccbe-0724-4170-bb40-76a54c5affc4" containerName="revoke-keys" Nov 02 20:09:25.675550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.675531 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700061 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e31a97-85a6-4372-9d49-9a59d5da22b4-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700093 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700134 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700207 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700316 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700351 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700468 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700512 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.700594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.700583 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnb7\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-kube-api-access-lsnb7\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.757762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.755544 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps"] Nov 02 20:09:25.802380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802344 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802428 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802479 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnb7\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-kube-api-access-lsnb7\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802589 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e31a97-85a6-4372-9d49-9a59d5da22b4-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802622 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802680 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802703 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802731 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802757 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.802859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802831 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.803072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.802875 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.803373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.803346 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.803495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.803479 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.803615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.803583 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e31a97-85a6-4372-9d49-9a59d5da22b4-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.805238 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.805219 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.805690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.805673 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.855520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.855480 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.855658 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.855596 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnb7\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-kube-api-access-lsnb7\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.990082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.989960 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:25.994223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.994191 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:09:25.994702 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:25.994254 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 20:09:26.458641 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:26.457951 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e31a97_85a6_4372_9d49_9a59d5da22b4.slice/crio-003f3688052486eccd0ed80fdd4a2206b35f90407c3b22209301c5ee4e4e8e5d WatchSource:0}: Error finding container 003f3688052486eccd0ed80fdd4a2206b35f90407c3b22209301c5ee4e4e8e5d: Status 404 returned error can't find the container with id 003f3688052486eccd0ed80fdd4a2206b35f90407c3b22209301c5ee4e4e8e5d Nov 02 20:09:26.460274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:26.460248 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps"] Nov 02 20:09:26.460955 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:26.460920 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:26.461047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:26.461003 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:26.461047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:26.461040 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:26.993944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:26.993904 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:09:26.994130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:26.993964 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 20:09:27.414075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.414039 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" event={"ID":"10e31a97-85a6-4372-9d49-9a59d5da22b4","Type":"ContainerStarted","Data":"e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7"} Nov 02 20:09:27.414520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.414134 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" event={"ID":"10e31a97-85a6-4372-9d49-9a59d5da22b4","Type":"ContainerStarted","Data":"003f3688052486eccd0ed80fdd4a2206b35f90407c3b22209301c5ee4e4e8e5d"} Nov 02 20:09:27.646304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.646059 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podStartSLOduration=2.646038156 podStartE2EDuration="2.646038156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:27.579546833 +0000 UTC m=+1266.096319423" watchObservedRunningTime="2026-11-02 20:09:27.646038156 +0000 UTC m=+1266.162810743" Nov 02 20:09:27.692724 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:27.692642 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:09:27.990919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.990834 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:27.992756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.992725 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 20:09:27.992895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.992785 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 20:09:27.993323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.993267 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 02 20:09:27.993434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:27.993339 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 02 20:09:28.991442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:28.991407 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 20:09:28.991930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:28.991465 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 20:09:28.998126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:28.998101 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:28.998540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:28.998524 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:28.999200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:28.999179 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:09:29.052412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.052371 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4"] Nov 02 20:09:29.090940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.090910 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a49c91-kxgc4"] Nov 02 20:09:29.622176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.622139 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0" path="/var/lib/kubelet/pods/fcd6d7ff-66ac-44a9-8dad-12f25d67bfd0/volumes" Nov 02 20:09:29.917915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.917688 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29"] Nov 02 20:09:29.924089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.924062 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:29.991926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.991891 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29"] Nov 02 20:09:29.998303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.996381 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 20:09:29.998303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:29.996453 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 20:09:30.018648 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.018615 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:09:30.018967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.018946 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:30.019221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.019204 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:09:30.019371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.019313 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:09:30.119002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.118970 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdk2r\" (UniqueName: \"kubernetes.io/projected/f3548f1c-0a36-4fb2-9775-675eb998acf4-kube-api-access-jdk2r\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.119159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.119109 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.119197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.119171 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.220846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.220758 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdk2r\" (UniqueName: \"kubernetes.io/projected/f3548f1c-0a36-4fb2-9775-675eb998acf4-kube-api-access-jdk2r\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.221004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.220865 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.221004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.220926 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.221621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.221597 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.223308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.223257 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.274312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.274250 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdk2r\" (UniqueName: \"kubernetes.io/projected/f3548f1c-0a36-4fb2-9775-675eb998acf4-kube-api-access-jdk2r\") pod \"maas-api-revoke-sub-3b67a77e-95j29\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.566581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.566539 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:30.985817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.985780 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29"] Nov 02 20:09:30.986356 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:30.986325 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3548f1c_0a36_4fb2_9775_675eb998acf4.slice/crio-0860ebd996663a6b0d6f0e585b80794bffffcab0e99ebd768c37c0dfa85940b1 WatchSource:0}: Error finding container 0860ebd996663a6b0d6f0e585b80794bffffcab0e99ebd768c37c0dfa85940b1: Status 404 returned error can't find the container with id 0860ebd996663a6b0d6f0e585b80794bffffcab0e99ebd768c37c0dfa85940b1 Nov 02 20:09:30.991273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.991252 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 20:09:30.991379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:30.991308 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 20:09:31.471080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:31.471037 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" event={"ID":"f3548f1c-0a36-4fb2-9775-675eb998acf4","Type":"ContainerStarted","Data":"0860ebd996663a6b0d6f0e585b80794bffffcab0e99ebd768c37c0dfa85940b1"} Nov 02 20:09:31.991671 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:31.991630 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 20:09:31.991853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:31.991698 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 20:09:32.482678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:32.482640 5800 generic.go:356] "Generic (PLEG): container finished" podID="f3548f1c-0a36-4fb2-9775-675eb998acf4" containerID="9e49ba5bd6533ca3af124df28cae86e3306e7f4795be9ab956cca03ca20798a0" exitCode=0 Nov 02 20:09:32.483118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:32.482683 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" event={"ID":"f3548f1c-0a36-4fb2-9775-675eb998acf4","Type":"ContainerDied","Data":"9e49ba5bd6533ca3af124df28cae86e3306e7f4795be9ab956cca03ca20798a0"} Nov 02 20:09:32.990731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:32.990695 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 20:09:32.990939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:32.990748 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 20:09:33.213301 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:33.204974 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:09:33.690013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.689985 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:33.712477 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.712453 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3548f1c-0a36-4fb2-9775-675eb998acf4-kube-api-access-jdk2r\" (UniqueName: \"kubernetes.io/projected/f3548f1c-0a36-4fb2-9775-675eb998acf4-kube-api-access-jdk2r\") pod \"f3548f1c-0a36-4fb2-9775-675eb998acf4\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " Nov 02 20:09:33.712622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.712504 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-openshift-ca\") pod \"f3548f1c-0a36-4fb2-9775-675eb998acf4\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " Nov 02 20:09:33.712697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.712679 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-xks-ca\") pod \"f3548f1c-0a36-4fb2-9775-675eb998acf4\" (UID: \"f3548f1c-0a36-4fb2-9775-675eb998acf4\") " Nov 02 20:09:33.712930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.712901 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-openshift-ca" pod "f3548f1c-0a36-4fb2-9775-675eb998acf4" (UID: "f3548f1c-0a36-4fb2-9775-675eb998acf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:33.713141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.713124 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:33.714492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.714467 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3548f1c-0a36-4fb2-9775-675eb998acf4-kube-api-access-jdk2r" pod "f3548f1c-0a36-4fb2-9775-675eb998acf4" (UID: "f3548f1c-0a36-4fb2-9775-675eb998acf4"). InnerVolumeSpecName "kube-api-access-jdk2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:33.714619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.714601 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-xks-ca" pod "f3548f1c-0a36-4fb2-9775-675eb998acf4" (UID: "f3548f1c-0a36-4fb2-9775-675eb998acf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:33.815256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.815215 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdk2r\" (UniqueName: \"kubernetes.io/projected/f3548f1c-0a36-4fb2-9775-675eb998acf4-kube-api-access-jdk2r\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:33.815427 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.815260 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3548f1c-0a36-4fb2-9775-675eb998acf4-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:33.995672 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.995639 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:33.995961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.995938 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:33.996616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:33.996598 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:09:34.094883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.094806 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27"] Nov 02 20:09:34.095417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.095403 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f3548f1c-0a36-4fb2-9775-675eb998acf4" containerName="revoke-keys" Nov 02 20:09:34.095466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.095417 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3548f1c-0a36-4fb2-9775-675eb998acf4" containerName="revoke-keys" Nov 02 20:09:34.095528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.095519 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3548f1c-0a36-4fb2-9775-675eb998acf4" containerName="revoke-keys" Nov 02 20:09:34.098997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.098979 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.118552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.118523 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvts\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-kube-api-access-gpvts\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.118689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.118646 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.118749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.118714 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.118749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.118742 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.118824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.118802 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.118868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.118833 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.118868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.118862 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.119037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.119020 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.119097 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.119052 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.172742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.172706 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27"] Nov 02 20:09:34.220762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.220728 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.220928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.220771 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.220928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.220796 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.220928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.220833 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.220928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.220878 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.220928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.220905 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.221193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.221029 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.221193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.221058 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.221193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.221171 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvts\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-kube-api-access-gpvts\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.221738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.221614 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.221738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.221636 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.221738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.221691 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.221975 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.221931 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.222223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.222195 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.224413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.224378 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.226803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.226779 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.280255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.280112 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvts\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-kube-api-access-gpvts\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.292317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.290145 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.413527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.413445 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:34.514213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.513471 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" Nov 02 20:09:34.514213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.513506 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29" event={"ID":"f3548f1c-0a36-4fb2-9775-675eb998acf4","Type":"ContainerDied","Data":"0860ebd996663a6b0d6f0e585b80794bffffcab0e99ebd768c37c0dfa85940b1"} Nov 02 20:09:34.514213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.513576 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0860ebd996663a6b0d6f0e585b80794bffffcab0e99ebd768c37c0dfa85940b1" Nov 02 20:09:34.845896 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:34.845853 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099c86c1_d83a_450d_8ae0_908bac2cc0d4.slice/crio-525f20bec04542176ea28575491901e9b4e290f44144a90a922984f1ed16f84f WatchSource:0}: Error finding container 525f20bec04542176ea28575491901e9b4e290f44144a90a922984f1ed16f84f: Status 404 returned error can't find the container with id 525f20bec04542176ea28575491901e9b4e290f44144a90a922984f1ed16f84f Nov 02 20:09:34.848003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.847981 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27"] Nov 02 20:09:34.848997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.848961 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:34.849073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.849044 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:34.849108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.849088 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:34.992249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.992215 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck"] Nov 02 20:09:34.997487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:34.997464 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.034325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.032261 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkv4\" (UniqueName: \"kubernetes.io/projected/8a2004c9-8d63-457d-95ff-1be18e241777-kube-api-access-nqkv4\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.034325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.032349 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.034325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.032533 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.049511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.048398 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck"] Nov 02 20:09:35.049511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.048546 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:09:35.049511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.048614 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:35.049511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.049019 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:09:35.049511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.049178 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:09:35.134303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.134191 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.134488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.134422 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.134561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.134519 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkv4\" (UniqueName: \"kubernetes.io/projected/8a2004c9-8d63-457d-95ff-1be18e241777-kube-api-access-nqkv4\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.135147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.135115 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.136991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.136963 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.187220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.187189 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkv4\" (UniqueName: \"kubernetes.io/projected/8a2004c9-8d63-457d-95ff-1be18e241777-kube-api-access-nqkv4\") pod \"maas-api-revoke-sub-e8cb5322-nzrck\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.313680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.313639 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:35.542820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.522477 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" event={"ID":"099c86c1-d83a-450d-8ae0-908bac2cc0d4","Type":"ContainerStarted","Data":"89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd"} Nov 02 20:09:35.542820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.522543 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" event={"ID":"099c86c1-d83a-450d-8ae0-908bac2cc0d4","Type":"ContainerStarted","Data":"525f20bec04542176ea28575491901e9b4e290f44144a90a922984f1ed16f84f"} Nov 02 20:09:35.657476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.657350 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podStartSLOduration=1.657325031 podStartE2EDuration="1.657325031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:35.64146157 +0000 UTC m=+1274.158234153" watchObservedRunningTime="2026-11-02 20:09:35.657325031 +0000 UTC m=+1274.174097616" Nov 02 20:09:35.691613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:35.691569 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck"] Nov 02 20:09:35.695531 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:35.695500 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2004c9_8d63_457d_95ff_1be18e241777.slice/crio-e6b9d40fba5b6b52100804391ec732badc52c1005a3fdb3b2827bc3ca7cd86f1 WatchSource:0}: Error finding container e6b9d40fba5b6b52100804391ec732badc52c1005a3fdb3b2827bc3ca7cd86f1: Status 404 returned error can't find the container with id e6b9d40fba5b6b52100804391ec732badc52c1005a3fdb3b2827bc3ca7cd86f1 Nov 02 20:09:36.414415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:36.414376 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:36.416628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:36.416596 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:36.416782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:36.416658 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:36.570269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:36.570220 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" event={"ID":"8a2004c9-8d63-457d-95ff-1be18e241777","Type":"ContainerStarted","Data":"1a9656d4fb4f27dc5f14b1e969ef2221a8b1cb8d78fdf7c6736bf014bb904ab5"} Nov 02 20:09:36.570686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:36.570661 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" event={"ID":"8a2004c9-8d63-457d-95ff-1be18e241777","Type":"ContainerStarted","Data":"e6b9d40fba5b6b52100804391ec732badc52c1005a3fdb3b2827bc3ca7cd86f1"} Nov 02 20:09:37.417308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:37.416856 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:37.417308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:37.416938 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:37.705363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:37.704480 5800 generic.go:356] "Generic (PLEG): container finished" podID="8a2004c9-8d63-457d-95ff-1be18e241777" containerID="1a9656d4fb4f27dc5f14b1e969ef2221a8b1cb8d78fdf7c6736bf014bb904ab5" exitCode=0 Nov 02 20:09:37.713303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:37.705851 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" event={"ID":"8a2004c9-8d63-457d-95ff-1be18e241777","Type":"ContainerDied","Data":"1a9656d4fb4f27dc5f14b1e969ef2221a8b1cb8d78fdf7c6736bf014bb904ab5"} Nov 02 20:09:37.754305 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:37.753842 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:09:38.330306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.329423 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:38.334304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.330768 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:38.415825 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.415780 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:38.416129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.416088 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:38.474713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.474655 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:38.475570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.475345 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 20:09:38.497504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.497387 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Nov 02 20:09:38.629412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.627478 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:38.674304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.673393 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:38.674304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:38.673946 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 20:09:39.184099 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.184059 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:39.184445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.184394 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.332508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.332444 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.333387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.333354 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:09:39.335983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.334872 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.335983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.335241 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 20:09:39.416798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.416759 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:39.417099 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.417055 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:39.486361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.481382 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.486361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.481864 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:09:39.486361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.483177 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.486361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.483541 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:09:39.634308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.630040 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.695304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.675401 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:39.695304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.675840 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 02 20:09:40.076308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:39.958353 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:09:40.160607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.160513 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:09:40.160859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.160841 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:09:40.398301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.391308 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 20:09:40.435420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.405421 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 02 20:09:40.510336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.481804 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:40.510336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.484444 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:40.510336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.510123 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:40.510336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.510181 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:40.546403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.534430 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:09:40.630481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.628379 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh"] Nov 02 20:09:40.630919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.630667 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:40.679440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:40.679061 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:41.187311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:41.125491 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:09:41.414308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:41.411649 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:41.414308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:41.159940 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f53506-n6xsh"] Nov 02 20:09:41.542408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:41.479199 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:42.181488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.181437 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:42.260312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.220209 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:42.301493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.301454 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:42.301717 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.291972 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:09:42.322053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.322001 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 02 20:09:42.340931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.340896 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e8d8c5-08da-4acf-bd63-a561b23292b2" path="/var/lib/kubelet/pods/76e8d8c5-08da-4acf-bd63-a561b23292b2/volumes" Nov 02 20:09:42.355719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.355202 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 20:09:42.357579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.356129 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:09:42.357579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.357519 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 02 20:09:42.358050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.357818 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 20:09:42.358180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.358141 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 20:09:42.396928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.396892 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:42.415098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.415029 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:42.415356 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.415329 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:42.448413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.426489 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-xks-ca\") pod \"8a2004c9-8d63-457d-95ff-1be18e241777\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " Nov 02 20:09:42.448413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.426616 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2004c9-8d63-457d-95ff-1be18e241777-kube-api-access-nqkv4\" (UniqueName: \"kubernetes.io/projected/8a2004c9-8d63-457d-95ff-1be18e241777-kube-api-access-nqkv4\") pod \"8a2004c9-8d63-457d-95ff-1be18e241777\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " Nov 02 20:09:42.448413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.426688 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-openshift-ca\") pod \"8a2004c9-8d63-457d-95ff-1be18e241777\" (UID: \"8a2004c9-8d63-457d-95ff-1be18e241777\") " Nov 02 20:09:42.452422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.452388 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:09:42.452663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.452650 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:09:42.482304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.467502 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-openshift-ca" pod "8a2004c9-8d63-457d-95ff-1be18e241777" (UID: "8a2004c9-8d63-457d-95ff-1be18e241777"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:42.482304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.476160 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" Nov 02 20:09:42.482304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.476332 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck" event={"ID":"8a2004c9-8d63-457d-95ff-1be18e241777","Type":"ContainerDied","Data":"e6b9d40fba5b6b52100804391ec732badc52c1005a3fdb3b2827bc3ca7cd86f1"} Nov 02 20:09:42.482304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.476375 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b9d40fba5b6b52100804391ec732badc52c1005a3fdb3b2827bc3ca7cd86f1" Nov 02 20:09:42.494366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.492337 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2004c9-8d63-457d-95ff-1be18e241777-kube-api-access-nqkv4" pod "8a2004c9-8d63-457d-95ff-1be18e241777" (UID: "8a2004c9-8d63-457d-95ff-1be18e241777"). InnerVolumeSpecName "kube-api-access-nqkv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:42.495470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.495436 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-xks-ca" pod "8a2004c9-8d63-457d-95ff-1be18e241777" (UID: "8a2004c9-8d63-457d-95ff-1be18e241777"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:42.495964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.495940 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:09:42.496945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.496922 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 20:09:42.522306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.522267 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:09:42.570022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.569964 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:42.570022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.570016 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqkv4\" (UniqueName: \"kubernetes.io/projected/8a2004c9-8d63-457d-95ff-1be18e241777-kube-api-access-nqkv4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:42.570633 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.570037 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2004c9-8d63-457d-95ff-1be18e241777-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:42.990083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.990042 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm"] Nov 02 20:09:42.990675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.990658 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8a2004c9-8d63-457d-95ff-1be18e241777" containerName="revoke-keys" Nov 02 20:09:42.990675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.990674 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2004c9-8d63-457d-95ff-1be18e241777" containerName="revoke-keys" Nov 02 20:09:42.990793 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.990779 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2004c9-8d63-457d-95ff-1be18e241777" containerName="revoke-keys" Nov 02 20:09:42.994402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:42.994381 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.054362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.052692 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:09:43.054362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.052991 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:09:43.054362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.053067 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:43.054362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.053201 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:09:43.080443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.080406 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xzkw\" (UniqueName: \"kubernetes.io/projected/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-kube-api-access-5xzkw\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.080632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.080471 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.080704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.080676 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.081112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.081085 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm"] Nov 02 20:09:43.183254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.182858 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.183254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.183095 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.183254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.183212 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xzkw\" (UniqueName: \"kubernetes.io/projected/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-kube-api-access-5xzkw\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.184319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.184257 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.187151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.187130 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.266517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.266486 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xzkw\" (UniqueName: \"kubernetes.io/projected/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-kube-api-access-5xzkw\") pod \"maas-api-revoke-sub-f1a97324-7lxfm\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.312583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.312491 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:43.413908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.413873 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:43.414111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.413930 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:43.767656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:43.767619 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm"] Nov 02 20:09:43.769903 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:43.769852 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d466fb2_0664_47b2_a51c_f8b40f26e5b6.slice/crio-301a914cc7dc021b23f0268bcb04974093afe21fc541799d5b816ef1a561cb47 WatchSource:0}: Error finding container 301a914cc7dc021b23f0268bcb04974093afe21fc541799d5b816ef1a561cb47: Status 404 returned error can't find the container with id 301a914cc7dc021b23f0268bcb04974093afe21fc541799d5b816ef1a561cb47 Nov 02 20:09:44.102989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.102909 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg"] Nov 02 20:09:44.108480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.108456 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.220394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.216588 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5e04421b-openshift-default-dockercfg-b5655\"" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.262560 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.262787 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.263074 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.263227 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.263354 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.263423 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.263490 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.263661 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.268302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.263792 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9wl\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-kube-api-access-hf9wl\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.307368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.287369 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg"] Nov 02 20:09:44.367490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.366621 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.368223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.368086 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9wl\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-kube-api-access-hf9wl\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.374598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.373609 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.374598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.373670 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.374598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.373735 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.374598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.373780 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.374598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.373873 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.374598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.373903 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.374598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.373933 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.375804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.375712 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.379158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.376217 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.379158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.378879 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.379158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.379122 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.379593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.379536 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.393716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.392725 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.393716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.393641 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.414061 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.414021 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:44.414233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.414100 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:44.429294 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.429233 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9wl\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-kube-api-access-hf9wl\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.431705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.431671 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:44.524460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.524424 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" event={"ID":"8d466fb2-0664-47b2-a51c-f8b40f26e5b6","Type":"ContainerStarted","Data":"32d1532fb54a4d511e777ec3959f8d0e5cbef0e77f7c052e152e8985cd2730d4"} Nov 02 20:09:44.524697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.524676 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" event={"ID":"8d466fb2-0664-47b2-a51c-f8b40f26e5b6","Type":"ContainerStarted","Data":"301a914cc7dc021b23f0268bcb04974093afe21fc541799d5b816ef1a561cb47"} Nov 02 20:09:44.639778 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.639671 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" podStartSLOduration=2.63965666 podStartE2EDuration="2.63965666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:44.636674676 +0000 UTC m=+1283.153447262" watchObservedRunningTime="2026-11-02 20:09:44.63965666 +0000 UTC m=+1283.156429244" Nov 02 20:09:44.722594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:44.722549 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:45.247848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.247818 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg"] Nov 02 20:09:45.248786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.248746 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:45.248897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.248826 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:45.248897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.248870 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:09:45.417405 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.417362 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 02 20:09:45.417606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.417428 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 02 20:09:45.555332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.555017 5800 generic.go:356] "Generic (PLEG): container finished" podID="8d466fb2-0664-47b2-a51c-f8b40f26e5b6" containerID="32d1532fb54a4d511e777ec3959f8d0e5cbef0e77f7c052e152e8985cd2730d4" exitCode=0 Nov 02 20:09:45.555332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.555217 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" event={"ID":"8d466fb2-0664-47b2-a51c-f8b40f26e5b6","Type":"ContainerDied","Data":"32d1532fb54a4d511e777ec3959f8d0e5cbef0e77f7c052e152e8985cd2730d4"} Nov 02 20:09:45.558587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.557727 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" event={"ID":"bb9856b5-90d2-4e34-84a7-43724bc9e1cd","Type":"ContainerStarted","Data":"99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685"} Nov 02 20:09:45.558587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.557784 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" event={"ID":"bb9856b5-90d2-4e34-84a7-43724bc9e1cd","Type":"ContainerStarted","Data":"3b9c6916b75b8d8aa4a4d8556bd169bd25a5369f245161d05f369000a660055f"} Nov 02 20:09:45.729536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.729492 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:45.795057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:45.795007 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podStartSLOduration=1.794992083 podStartE2EDuration="1.794992083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:45.793017761 +0000 UTC m=+1284.309790346" watchObservedRunningTime="2026-11-02 20:09:45.794992083 +0000 UTC m=+1284.311764668" Nov 02 20:09:46.737208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.737166 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:46.758788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.758760 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:46.913849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.913819 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-xks-ca\") pod \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " Nov 02 20:09:46.914040 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.914021 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-openshift-ca\") pod \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " Nov 02 20:09:46.914182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.914168 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-kube-api-access-5xzkw\" (UniqueName: \"kubernetes.io/projected/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-kube-api-access-5xzkw\") pod \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\" (UID: \"8d466fb2-0664-47b2-a51c-f8b40f26e5b6\") " Nov 02 20:09:46.914466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.914426 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-openshift-ca" pod "8d466fb2-0664-47b2-a51c-f8b40f26e5b6" (UID: "8d466fb2-0664-47b2-a51c-f8b40f26e5b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:09:46.914616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.914598 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:46.915924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.915886 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-xks-ca" pod "8d466fb2-0664-47b2-a51c-f8b40f26e5b6" (UID: "8d466fb2-0664-47b2-a51c-f8b40f26e5b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:09:46.916147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:46.916126 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-kube-api-access-5xzkw" pod "8d466fb2-0664-47b2-a51c-f8b40f26e5b6" (UID: "8d466fb2-0664-47b2-a51c-f8b40f26e5b6"). InnerVolumeSpecName "kube-api-access-5xzkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:09:47.016776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.016735 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xzkw\" (UniqueName: \"kubernetes.io/projected/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-kube-api-access-5xzkw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:47.016776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.016777 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d466fb2-0664-47b2-a51c-f8b40f26e5b6-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:09:47.240211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.240125 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:47.240547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.240524 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:47.241470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.241442 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:09:47.584262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.584212 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" event={"ID":"8d466fb2-0664-47b2-a51c-f8b40f26e5b6","Type":"ContainerDied","Data":"301a914cc7dc021b23f0268bcb04974093afe21fc541799d5b816ef1a561cb47"} Nov 02 20:09:47.584495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.584293 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="301a914cc7dc021b23f0268bcb04974093afe21fc541799d5b816ef1a561cb47" Nov 02 20:09:47.584495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.584236 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm" Nov 02 20:09:47.584958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.584935 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:47.586630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:47.586600 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:09:47.863163 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:47.863080 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:09:48.184856 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:48.184768 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:09:52.332175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.332138 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk"] Nov 02 20:09:52.332772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.332750 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8d466fb2-0664-47b2-a51c-f8b40f26e5b6" containerName="revoke-keys" Nov 02 20:09:52.332772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.332770 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d466fb2-0664-47b2-a51c-f8b40f26e5b6" containerName="revoke-keys" Nov 02 20:09:52.332994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.332980 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d466fb2-0664-47b2-a51c-f8b40f26e5b6" containerName="revoke-keys" Nov 02 20:09:52.337958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.337933 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:52.389946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.389915 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk"] Nov 02 20:09:52.444709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.444552 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5e04421b\"" Nov 02 20:09:52.448994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.445199 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5e04421b-dockercfg-qsbhl\"" Nov 02 20:09:52.531730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.531670 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b719543-4c07-4e26-ab54-a1e1483deb2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:52.532147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.532114 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzp8k\" (UniqueName: \"kubernetes.io/projected/8b719543-4c07-4e26-ab54-a1e1483deb2e-kube-api-access-lzp8k\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:52.634040 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.633953 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzp8k\" (UniqueName: \"kubernetes.io/projected/8b719543-4c07-4e26-ab54-a1e1483deb2e-kube-api-access-lzp8k\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:52.634230 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.634100 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b719543-4c07-4e26-ab54-a1e1483deb2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:52.634804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.634786 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b719543-4c07-4e26-ab54-a1e1483deb2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:52.686908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.686880 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzp8k\" (UniqueName: \"kubernetes.io/projected/8b719543-4c07-4e26-ab54-a1e1483deb2e-kube-api-access-lzp8k\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:52.697074 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:52.697041 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:09:52.954322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:52.954232 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:53.352420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:53.352391 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk"] Nov 02 20:09:53.353572 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:53.353541 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b719543_4c07_4e26_ab54_a1e1483deb2e.slice/crio-e534d7686fc12870f28f6bd27616f1f93cf099ef7c2119b6c1e0fb2cb1e703f7 WatchSource:0}: Error finding container e534d7686fc12870f28f6bd27616f1f93cf099ef7c2119b6c1e0fb2cb1e703f7: Status 404 returned error can't find the container with id e534d7686fc12870f28f6bd27616f1f93cf099ef7c2119b6c1e0fb2cb1e703f7 Nov 02 20:09:53.637868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:53.637771 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" event={"ID":"8b719543-4c07-4e26-ab54-a1e1483deb2e","Type":"ContainerStarted","Data":"1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175"} Nov 02 20:09:53.637868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:53.637829 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" event={"ID":"8b719543-4c07-4e26-ab54-a1e1483deb2e","Type":"ContainerStarted","Data":"e534d7686fc12870f28f6bd27616f1f93cf099ef7c2119b6c1e0fb2cb1e703f7"} Nov 02 20:09:53.637868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:53.637860 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:09:53.747355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:53.747304 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" podStartSLOduration=1.747273981 podStartE2EDuration="1.747273981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:53.742408447 +0000 UTC m=+1292.259181034" watchObservedRunningTime="2026-11-02 20:09:53.747273981 +0000 UTC m=+1292.264046565" Nov 02 20:09:56.605153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.605109 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-94m58"] Nov 02 20:09:56.610651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.610628 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.660347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.660307 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-94m58"] Nov 02 20:09:56.689659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.689615 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-routing-overlay-volume\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.689846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.689788 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxc8z\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-kube-api-access-qxc8z\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.690000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.689981 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-plugins-config-volume\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.690079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.690037 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-provider-credentials\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.791378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.791341 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxc8z\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-kube-api-access-qxc8z\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.791588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.791482 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-plugins-config-volume\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.791588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.791518 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-provider-credentials\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.791705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.791620 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-routing-overlay-volume\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.792376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.792348 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-plugins-config-volume\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.792495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.792353 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-routing-overlay-volume\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.793983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.793962 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-provider-credentials\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.844260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.844218 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxc8z\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-kube-api-access-qxc8z\") pod \"payload-processing-external-model-cd9f598dd-94m58\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.915814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.915739 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4"] Nov 02 20:09:56.920701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.920680 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:56.925009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.924986 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:56.970889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.970858 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:09:56.971636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.971356 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:09:56.973573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.972349 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:09:56.973573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.972453 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:09:56.974673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.974648 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4"] Nov 02 20:09:56.995489 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.995452 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cc0532-f285-4081-9e42-11c206442d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:56.995702 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.995635 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cc0532-f285-4081-9e42-11c206442d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:56.995819 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:56.995735 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr6m9\" (UniqueName: \"kubernetes.io/projected/46cc0532-f285-4081-9e42-11c206442d17-kube-api-access-hr6m9\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.118307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.110805 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cc0532-f285-4081-9e42-11c206442d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.118307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.112392 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cc0532-f285-4081-9e42-11c206442d17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.118307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.113538 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cc0532-f285-4081-9e42-11c206442d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.118307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.113677 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr6m9\" (UniqueName: \"kubernetes.io/projected/46cc0532-f285-4081-9e42-11c206442d17-kube-api-access-hr6m9\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.203783 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.203680 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr6m9\" (UniqueName: \"kubernetes.io/projected/46cc0532-f285-4081-9e42-11c206442d17-kube-api-access-hr6m9\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.205625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.205576 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cc0532-f285-4081-9e42-11c206442d17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90fb2618-s29b4\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.237544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.237510 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:09:57.329513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.329021 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-94m58"] Nov 02 20:09:57.333357 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:57.333261 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97afc77_9f58_4c30_8cc0_ced98cef958f.slice/crio-3dd51e9548eff76ce252086ba3158926ca84ba51d7fe1c0ffb7fa903dce81b16 WatchSource:0}: Error finding container 3dd51e9548eff76ce252086ba3158926ca84ba51d7fe1c0ffb7fa903dce81b16: Status 404 returned error can't find the container with id 3dd51e9548eff76ce252086ba3158926ca84ba51d7fe1c0ffb7fa903dce81b16 Nov 02 20:09:57.615140 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:09:57.615105 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cc0532_f285_4081_9e42_11c206442d17.slice/crio-b1d716591e21c4c7492b841ca54c5c23df23d80b813dd2ffe07dbda8f0135acd WatchSource:0}: Error finding container b1d716591e21c4c7492b841ca54c5c23df23d80b813dd2ffe07dbda8f0135acd: Status 404 returned error can't find the container with id b1d716591e21c4c7492b841ca54c5c23df23d80b813dd2ffe07dbda8f0135acd Nov 02 20:09:57.624232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.624191 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4"] Nov 02 20:09:57.678245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.678214 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" event={"ID":"46cc0532-f285-4081-9e42-11c206442d17","Type":"ContainerStarted","Data":"b1d716591e21c4c7492b841ca54c5c23df23d80b813dd2ffe07dbda8f0135acd"} Nov 02 20:09:57.679907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.679877 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" event={"ID":"e97afc77-9f58-4c30-8cc0-ced98cef958f","Type":"ContainerStarted","Data":"057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda"} Nov 02 20:09:57.680010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.679929 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" event={"ID":"e97afc77-9f58-4c30-8cc0-ced98cef958f","Type":"ContainerStarted","Data":"3dd51e9548eff76ce252086ba3158926ca84ba51d7fe1c0ffb7fa903dce81b16"} Nov 02 20:09:57.680010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.679969 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:09:57.788894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:57.788834 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" podStartSLOduration=1.788819119 podStartE2EDuration="1.788819119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:57.786491187 +0000 UTC m=+1296.303263772" watchObservedRunningTime="2026-11-02 20:09:57.788819119 +0000 UTC m=+1296.305591738" Nov 02 20:09:57.881341 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:09:57.881253 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:09:58.692165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:58.692129 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" event={"ID":"46cc0532-f285-4081-9e42-11c206442d17","Type":"ContainerStarted","Data":"5facfef97e52763e0f92628c6b927494d6bb9f468e148a9a660615ca47c6cc01"} Nov 02 20:09:58.801669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:58.801611 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" podStartSLOduration=2.801593393 podStartE2EDuration="2.801593393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:09:58.797125168 +0000 UTC m=+1297.313897756" watchObservedRunningTime="2026-11-02 20:09:58.801593393 +0000 UTC m=+1297.318365979" Nov 02 20:09:59.050120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:59.050082 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52"] Nov 02 20:09:59.056209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:59.056179 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd558761-qfb52"] Nov 02 20:09:59.629742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:59.629706 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95f78e2-e9d7-450f-afb4-144cdaa5ab28" path="/var/lib/kubelet/pods/f95f78e2-e9d7-450f-afb4-144cdaa5ab28/volumes" Nov 02 20:09:59.700902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:59.700867 5800 generic.go:356] "Generic (PLEG): container finished" podID="46cc0532-f285-4081-9e42-11c206442d17" containerID="5facfef97e52763e0f92628c6b927494d6bb9f468e148a9a660615ca47c6cc01" exitCode=0 Nov 02 20:09:59.701275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:09:59.700953 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" event={"ID":"46cc0532-f285-4081-9e42-11c206442d17","Type":"ContainerDied","Data":"5facfef97e52763e0f92628c6b927494d6bb9f468e148a9a660615ca47c6cc01"} Nov 02 20:10:00.894981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.894955 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:10:00.969198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.969093 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cc0532-f285-4081-9e42-11c206442d17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cc0532-f285-4081-9e42-11c206442d17-maas-api-openshift-ca\") pod \"46cc0532-f285-4081-9e42-11c206442d17\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " Nov 02 20:10:00.969405 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.969358 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cc0532-f285-4081-9e42-11c206442d17-kube-api-access-hr6m9\" (UniqueName: \"kubernetes.io/projected/46cc0532-f285-4081-9e42-11c206442d17-kube-api-access-hr6m9\") pod \"46cc0532-f285-4081-9e42-11c206442d17\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " Nov 02 20:10:00.969493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.969472 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46cc0532-f285-4081-9e42-11c206442d17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cc0532-f285-4081-9e42-11c206442d17-maas-api-xks-ca\") pod \"46cc0532-f285-4081-9e42-11c206442d17\" (UID: \"46cc0532-f285-4081-9e42-11c206442d17\") " Nov 02 20:10:00.969558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.969523 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cc0532-f285-4081-9e42-11c206442d17-maas-api-openshift-ca" pod "46cc0532-f285-4081-9e42-11c206442d17" (UID: "46cc0532-f285-4081-9e42-11c206442d17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:00.970024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.970004 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46cc0532-f285-4081-9e42-11c206442d17-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:00.972039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.972004 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cc0532-f285-4081-9e42-11c206442d17-maas-api-xks-ca" pod "46cc0532-f285-4081-9e42-11c206442d17" (UID: "46cc0532-f285-4081-9e42-11c206442d17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:10:00.972151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:00.972063 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cc0532-f285-4081-9e42-11c206442d17-kube-api-access-hr6m9" pod "46cc0532-f285-4081-9e42-11c206442d17" (UID: "46cc0532-f285-4081-9e42-11c206442d17"). InnerVolumeSpecName "kube-api-access-hr6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:01.071830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:01.071777 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr6m9\" (UniqueName: \"kubernetes.io/projected/46cc0532-f285-4081-9e42-11c206442d17-kube-api-access-hr6m9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:01.071830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:01.071829 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46cc0532-f285-4081-9e42-11c206442d17-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:01.718581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:01.718555 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" Nov 02 20:10:01.718759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:01.718586 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4" event={"ID":"46cc0532-f285-4081-9e42-11c206442d17","Type":"ContainerDied","Data":"b1d716591e21c4c7492b841ca54c5c23df23d80b813dd2ffe07dbda8f0135acd"} Nov 02 20:10:01.718759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:01.718634 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d716591e21c4c7492b841ca54c5c23df23d80b813dd2ffe07dbda8f0135acd" Nov 02 20:10:03.455330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:03.453387 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:03.490308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:03.489379 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:03.533314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:03.532379 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:03.562546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:03.560419 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:03.564870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:03.564823 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:03.662320 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:03.648990 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:10:04.212306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.207413 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.338319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.333374 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.456324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.454375 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.456324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.455173 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:10:04.495397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.491379 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.495397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.491825 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:10:04.533319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.530398 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.546309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.544993 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.546309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.545390 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:10:04.568307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.563634 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.568307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.563898 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:10:04.568591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.568525 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:04.569955 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.568835 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:10:04.861313 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:04.845498 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:10:04.907788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:04.907739 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:10:05.031947 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:05.031899 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:10:05.078587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.078547 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded" Nov 02 20:10:05.358437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.349415 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.359821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.359792 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 20:10:05.582228 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.553575 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 02 20:10:05.593838 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.593795 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:05.605437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.605386 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.627505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.614508 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.652579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.612610 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 20:10:05.681307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.615589 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.743104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.612572 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.743104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.615637 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.743104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.704988 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:05.743104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.705032 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:05.980364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:05.961008 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:10:06.388683 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:06.388636 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 02 20:10:06.609629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:06.609575 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.713310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:06.713193 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 20:10:06.721343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:06.714369 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.727302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:06.726619 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded" Nov 02 20:10:06.738304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:06.736869 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 20:10:06.738304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:06.715519 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:06.972427 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:06.968117 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0188ccd53d43: secret "agc-cred-0188ccd53d43" not found Nov 02 20:10:06.972427 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:06.968173 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b94811517373: secret "agc-cred-b94811517373" not found Nov 02 20:10:06.972427 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:06.968190 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4: [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:06.972427 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:06.968321 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials podName:1d008eda-79f5-4bce-a0e6-40ce42d67ee5 nodeName:}" failed. No retries permitted until 2026-11-02 20:10:07.468297245 +0000 UTC m=+1305.985069830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials") pod "payload-processing-external-model-6d54dc9bd-5rrc4" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5") : [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:07.436647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:07.436598 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:07.464033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:07.463989 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:07.547312 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:07.530033 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0188ccd53d43: secret "agc-cred-0188ccd53d43" not found Nov 02 20:10:07.550044 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:07.550016 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b94811517373: secret "agc-cred-b94811517373" not found Nov 02 20:10:07.550229 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:07.550214 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4: [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:07.550437 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:07.550421 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials podName:1d008eda-79f5-4bce-a0e6-40ce42d67ee5 nodeName:}" failed. No retries permitted until 2026-11-02 20:10:08.55039688 +0000 UTC m=+1307.067169464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials") pod "payload-processing-external-model-6d54dc9bd-5rrc4" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5") : [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:07.647358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:07.647314 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 02 20:10:07.714497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:07.714393 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:07.772464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:07.772414 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:07.773078 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:07.773044 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 20:10:07.790991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:07.790949 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 20:10:08.186311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.167435 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 20:10:08.387873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.387841 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 20:10:08.420927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.420892 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 20:10:08.425679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.425651 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 20:10:08.472374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.439177 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:08.474844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.456450 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 20:10:08.494969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.494859 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:08.499202 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.497548 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" Nov 02 20:10:08.571823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.571792 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 20:10:08.648414 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:08.648383 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0188ccd53d43: secret "agc-cred-0188ccd53d43" not found Nov 02 20:10:08.648959 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:08.648940 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b94811517373: secret "agc-cred-b94811517373" not found Nov 02 20:10:08.649060 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:08.649050 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4: [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:08.649214 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:08.649200 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials podName:1d008eda-79f5-4bce-a0e6-40ce42d67ee5 nodeName:}" failed. No retries permitted until 2026-11-02 20:10:10.649179066 +0000 UTC m=+1309.165951646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials") pod "payload-processing-external-model-6d54dc9bd-5rrc4" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5") : [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:08.700704 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:08.700667 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:10:08.704499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:08.704465 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" podUID="e97afc77-9f58-4c30-8cc0-ced98cef958f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:10:09.510577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:09.510537 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-94m58"] Nov 02 20:10:09.513811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:09.512791 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" podUID="e97afc77-9f58-4c30-8cc0-ced98cef958f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:10:09.517801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:09.517774 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4"] Nov 02 20:10:10.687555 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:10.687521 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0188ccd53d43: secret "agc-cred-0188ccd53d43" not found Nov 02 20:10:10.687555 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:10.687554 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-543f396ff8c5: secret "agc-cred-543f396ff8c5" not found Nov 02 20:10:10.687555 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:10.687564 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b94811517373: secret "agc-cred-b94811517373" not found Nov 02 20:10:10.688030 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:10.687577 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4: [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-543f396ff8c5" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:10.688030 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:10.687645 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials podName:1d008eda-79f5-4bce-a0e6-40ce42d67ee5 nodeName:}" failed. No retries permitted until 2026-11-02 20:10:14.687624573 +0000 UTC m=+1313.204397137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials") pod "payload-processing-external-model-6d54dc9bd-5rrc4" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5") : [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-543f396ff8c5" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:14.511803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.511753 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" podUID="e97afc77-9f58-4c30-8cc0-ced98cef958f" containerName="payload-processing" containerID="cri-o://057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda" gracePeriod=25 Nov 02 20:10:14.518192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.518144 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" podUID="1d008eda-79f5-4bce-a0e6-40ce42d67ee5" containerName="payload-processing" containerID="cri-o://f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81" gracePeriod=25 Nov 02 20:10:14.743435 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:14.743249 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-0188ccd53d43: secret "agc-cred-0188ccd53d43" not found Nov 02 20:10:14.743435 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:14.743277 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-543f396ff8c5: secret "agc-cred-543f396ff8c5" not found Nov 02 20:10:14.743435 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:14.743305 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b94811517373: secret "agc-cred-b94811517373" not found Nov 02 20:10:14.743435 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:14.743317 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4: [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-543f396ff8c5" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:14.743435 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:14.743395 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials podName:1d008eda-79f5-4bce-a0e6-40ce42d67ee5 nodeName:}" failed. No retries permitted until 2026-11-02 20:10:22.743373834 +0000 UTC m=+1321.260146401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials") pod "payload-processing-external-model-6d54dc9bd-5rrc4" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5") : [secret "agc-cred-0188ccd53d43" not found, secret "agc-cred-543f396ff8c5" not found, secret "agc-cred-b94811517373" not found] Nov 02 20:10:14.838893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.838859 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:10:14.842956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.842930 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:10:14.945010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.944970 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-kube-api-access-qxc8z\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-kube-api-access-qxc8z\") pod \"e97afc77-9f58-4c30-8cc0-ced98cef958f\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " Nov 02 20:10:14.945186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945060 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-plugins-config-volume\") pod \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " Nov 02 20:10:14.945186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945104 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-routing-overlay-volume\") pod \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " Nov 02 20:10:14.945186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945151 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-plugins-config-volume\") pod \"e97afc77-9f58-4c30-8cc0-ced98cef958f\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " Nov 02 20:10:14.945439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945208 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-routing-overlay-volume\") pod \"e97afc77-9f58-4c30-8cc0-ced98cef958f\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " Nov 02 20:10:14.945439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945377 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials\") pod \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " Nov 02 20:10:14.945550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945439 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-provider-credentials\") pod \"e97afc77-9f58-4c30-8cc0-ced98cef958f\" (UID: \"e97afc77-9f58-4c30-8cc0-ced98cef958f\") " Nov 02 20:10:14.945599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945546 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-routing-overlay-volume" pod "1d008eda-79f5-4bce-a0e6-40ce42d67ee5" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:14.945599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945558 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-kube-api-access-7kqnm\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-kube-api-access-7kqnm\") pod \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\" (UID: \"1d008eda-79f5-4bce-a0e6-40ce42d67ee5\") " Nov 02 20:10:14.945697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945626 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-plugins-config-volume" pod "e97afc77-9f58-4c30-8cc0-ced98cef958f" (UID: "e97afc77-9f58-4c30-8cc0-ced98cef958f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:14.945697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.945634 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-plugins-config-volume" pod "1d008eda-79f5-4bce-a0e6-40ce42d67ee5" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:14.946499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.946444 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:14.946499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.946469 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:14.946499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.946463 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-routing-overlay-volume" pod "e97afc77-9f58-4c30-8cc0-ced98cef958f" (UID: "e97afc77-9f58-4c30-8cc0-ced98cef958f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:14.946739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.946484 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:14.948788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.948758 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-kube-api-access-7kqnm" pod "1d008eda-79f5-4bce-a0e6-40ce42d67ee5" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5"). InnerVolumeSpecName "kube-api-access-7kqnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:14.948943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.948786 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-kube-api-access-qxc8z" pod "e97afc77-9f58-4c30-8cc0-ced98cef958f" (UID: "e97afc77-9f58-4c30-8cc0-ced98cef958f"). InnerVolumeSpecName "kube-api-access-qxc8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:14.949036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.949013 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials" pod "1d008eda-79f5-4bce-a0e6-40ce42d67ee5" (UID: "1d008eda-79f5-4bce-a0e6-40ce42d67ee5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:14.949226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:14.949198 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-provider-credentials" pod "e97afc77-9f58-4c30-8cc0-ced98cef958f" (UID: "e97afc77-9f58-4c30-8cc0-ced98cef958f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:15.047813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.047730 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e97afc77-9f58-4c30-8cc0-ced98cef958f-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:15.047813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.047765 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:15.047813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.047783 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:15.047813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.047798 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kqnm\" (UniqueName: \"kubernetes.io/projected/1d008eda-79f5-4bce-a0e6-40ce42d67ee5-kube-api-access-7kqnm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:15.047813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.047813 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxc8z\" (UniqueName: \"kubernetes.io/projected/e97afc77-9f58-4c30-8cc0-ced98cef958f-kube-api-access-qxc8z\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:15.259001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.258966 5800 generic.go:356] "Generic (PLEG): container finished" podID="1d008eda-79f5-4bce-a0e6-40ce42d67ee5" containerID="f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81" exitCode=0 Nov 02 20:10:15.259180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.259016 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" Nov 02 20:10:15.259180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.259051 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" event={"ID":"1d008eda-79f5-4bce-a0e6-40ce42d67ee5","Type":"ContainerDied","Data":"f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81"} Nov 02 20:10:15.259180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.259105 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4" event={"ID":"1d008eda-79f5-4bce-a0e6-40ce42d67ee5","Type":"ContainerDied","Data":"9623ef54c7a942a286b8418c41bd0fccec806b25148e58e26068b29fa7447f2a"} Nov 02 20:10:15.259180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.259135 5800 scope.go:122] "RemoveContainer" containerID="f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81" Nov 02 20:10:15.260778 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.260749 5800 generic.go:356] "Generic (PLEG): container finished" podID="e97afc77-9f58-4c30-8cc0-ced98cef958f" containerID="057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda" exitCode=0 Nov 02 20:10:15.260895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.260780 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" event={"ID":"e97afc77-9f58-4c30-8cc0-ced98cef958f","Type":"ContainerDied","Data":"057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda"} Nov 02 20:10:15.260895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.260808 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" Nov 02 20:10:15.260895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.260844 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd9f598dd-94m58" event={"ID":"e97afc77-9f58-4c30-8cc0-ced98cef958f","Type":"ContainerDied","Data":"3dd51e9548eff76ce252086ba3158926ca84ba51d7fe1c0ffb7fa903dce81b16"} Nov 02 20:10:15.272500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.272482 5800 scope.go:122] "RemoveContainer" containerID="f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81" Nov 02 20:10:15.272821 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:15.272801 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81\": container with ID starting with f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81 not found: ID does not exist" containerID="f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81" Nov 02 20:10:15.272889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.272829 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81"} err="failed to get container status \"f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81\": rpc error: code = NotFound desc = could not find container \"f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81\": container with ID starting with f74185c0d08ac935765083703abab196c8265371cf6720b6c7a177491a90dc81 not found: ID does not exist" Nov 02 20:10:15.272889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.272846 5800 scope.go:122] "RemoveContainer" containerID="057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda" Nov 02 20:10:15.283072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.283050 5800 scope.go:122] "RemoveContainer" containerID="057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda" Nov 02 20:10:15.283410 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:15.283386 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda\": container with ID starting with 057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda not found: ID does not exist" containerID="057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda" Nov 02 20:10:15.283480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.283421 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda"} err="failed to get container status \"057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda\": rpc error: code = NotFound desc = could not find container \"057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda\": container with ID starting with 057b0c0edd5639f6659d11d19cebd168829b4eb1e9658692425175627f7f9dda not found: ID does not exist" Nov 02 20:10:15.418123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.418037 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4"] Nov 02 20:10:15.421652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.421620 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d54dc9bd-5rrc4"] Nov 02 20:10:15.525311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.524746 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-94m58"] Nov 02 20:10:15.530993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.530960 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd9f598dd-94m58"] Nov 02 20:10:15.623702 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.623671 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d008eda-79f5-4bce-a0e6-40ce42d67ee5" path="/var/lib/kubelet/pods/1d008eda-79f5-4bce-a0e6-40ce42d67ee5/volumes" Nov 02 20:10:15.624114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:15.624101 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97afc77-9f58-4c30-8cc0-ced98cef958f" path="/var/lib/kubelet/pods/e97afc77-9f58-4c30-8cc0-ced98cef958f/volumes" Nov 02 20:10:18.212812 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:18.212767 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:10:18.721202 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:18.721168 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877ccbe_0724_4170_bb40_76a54c5affc4.slice/crio-4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:10:21.174825 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.174785 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv"] Nov 02 20:10:21.175390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175375 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e97afc77-9f58-4c30-8cc0-ced98cef958f" containerName="payload-processing" Nov 02 20:10:21.175390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175389 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97afc77-9f58-4c30-8cc0-ced98cef958f" containerName="payload-processing" Nov 02 20:10:21.175527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175434 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1d008eda-79f5-4bce-a0e6-40ce42d67ee5" containerName="payload-processing" Nov 02 20:10:21.175527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175440 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d008eda-79f5-4bce-a0e6-40ce42d67ee5" containerName="payload-processing" Nov 02 20:10:21.175527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175451 5800 cpu_manager.go:414] "containerMap: removing container" podUID="46cc0532-f285-4081-9e42-11c206442d17" containerName="revoke-keys" Nov 02 20:10:21.175527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175456 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cc0532-f285-4081-9e42-11c206442d17" containerName="revoke-keys" Nov 02 20:10:21.175723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175546 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d008eda-79f5-4bce-a0e6-40ce42d67ee5" containerName="payload-processing" Nov 02 20:10:21.175723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175554 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cc0532-f285-4081-9e42-11c206442d17" containerName="revoke-keys" Nov 02 20:10:21.175723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.175562 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97afc77-9f58-4c30-8cc0-ced98cef958f" containerName="payload-processing" Nov 02 20:10:21.180695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.180674 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236175 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236213 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236240 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236371 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236536 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg54n\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-kube-api-access-hg54n\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236578 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236672 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236709 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.236777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.236728 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.283240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.283203 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv"] Nov 02 20:10:21.337911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.337877 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg54n\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-kube-api-access-hg54n\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.337923 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.337962 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.337984 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.338002 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.338046 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.338062 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.338094 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.338203 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.338581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.338542 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.339233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.339002 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.339233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.339147 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.341469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.339634 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.343892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.343867 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.355984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.355929 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.357021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.356994 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.392611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.392576 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.392761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.392741 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg54n\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-kube-api-access-hg54n\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.498882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.498804 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:21.568686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.567973 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j"] Nov 02 20:10:21.583469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.583448 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.634048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.634010 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:10:21.634963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.634929 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:10:21.635236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.635220 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:10:21.635420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.635406 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:10:21.637668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.637642 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j"] Nov 02 20:10:21.642879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.642850 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.643082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.643064 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.643534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.643372 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gllv\" (UniqueName: \"kubernetes.io/projected/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-kube-api-access-7gllv\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.744638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.744605 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.744846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.744682 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.744918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.744853 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gllv\" (UniqueName: \"kubernetes.io/projected/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-kube-api-access-7gllv\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.745447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.745419 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.747434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.747408 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.797654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.797625 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gllv\" (UniqueName: \"kubernetes.io/projected/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-kube-api-access-7gllv\") pod \"maas-api-revoke-sub-0deb60b3-cbw6j\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.901786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.901758 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:21.923184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.923154 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv"] Nov 02 20:10:21.923754 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:10:21.923726 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4448bd32_9a2a_4cfd_9dd2_ee343df34df5.slice/crio-15b941693c19393dbe333e7e9011d670a221a24fcc146f9f987ed12bd8fd686c WatchSource:0}: Error finding container 15b941693c19393dbe333e7e9011d670a221a24fcc146f9f987ed12bd8fd686c: Status 404 returned error can't find the container with id 15b941693c19393dbe333e7e9011d670a221a24fcc146f9f987ed12bd8fd686c Nov 02 20:10:21.926534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.926492 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:21.926623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.926582 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:21.926665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:21.926621 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:22.058914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.058832 5800 scope.go:122] "RemoveContainer" containerID="1a31497d2d0beb7dadd602b6dda2c75f6e0db884fa3dfa20e4c6cb5aa50b3e68" Nov 02 20:10:22.077054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.077022 5800 scope.go:122] "RemoveContainer" containerID="84ca21d14c8a8ae00c2dd12f206466c1602a08e9e71960f1c52604adecdc70ad" Nov 02 20:10:22.090966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.090933 5800 scope.go:122] "RemoveContainer" containerID="2d69e4d4c359f2c77bea5d77262ca8103897bce740357076b410d90fc0079284" Nov 02 20:10:22.103412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.103389 5800 scope.go:122] "RemoveContainer" containerID="5a620c64425eb17dfe86b1e259f38e833fae82f30d6c4e6be553e04c1412fcf1" Nov 02 20:10:22.283631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.283600 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j"] Nov 02 20:10:22.284065 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:10:22.284027 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0722c5_b76a_4ad1_85aa_cabf1f2ca97e.slice/crio-3f6084912c6e6e5d39ac0669cbce121f9c55d7f09df78b99452ba9c3f3be9b98 WatchSource:0}: Error finding container 3f6084912c6e6e5d39ac0669cbce121f9c55d7f09df78b99452ba9c3f3be9b98: Status 404 returned error can't find the container with id 3f6084912c6e6e5d39ac0669cbce121f9c55d7f09df78b99452ba9c3f3be9b98 Nov 02 20:10:22.321599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.321507 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" event={"ID":"4448bd32-9a2a-4cfd-9dd2-ee343df34df5","Type":"ContainerStarted","Data":"c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df"} Nov 02 20:10:22.322026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.321858 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" event={"ID":"4448bd32-9a2a-4cfd-9dd2-ee343df34df5","Type":"ContainerStarted","Data":"15b941693c19393dbe333e7e9011d670a221a24fcc146f9f987ed12bd8fd686c"} Nov 02 20:10:22.323648 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.323626 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" event={"ID":"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e","Type":"ContainerStarted","Data":"3f6084912c6e6e5d39ac0669cbce121f9c55d7f09df78b99452ba9c3f3be9b98"} Nov 02 20:10:22.442452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.442387 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podStartSLOduration=1.442366883 podStartE2EDuration="1.442366883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:22.434741409 +0000 UTC m=+1320.951513994" watchObservedRunningTime="2026-11-02 20:10:22.442366883 +0000 UTC m=+1320.959139468" Nov 02 20:10:22.499092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:22.499058 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:22.696336 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:22.696196 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:10:23.358629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:23.358588 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" event={"ID":"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e","Type":"ContainerStarted","Data":"5a630d85374ffd2f0dc4253cd150601cde71e592cf2377820b1916bbd8936c28"} Nov 02 20:10:23.467201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:23.467142 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" podStartSLOduration=2.46712425 podStartE2EDuration="2.46712425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:23.464432477 +0000 UTC m=+1321.981205062" watchObservedRunningTime="2026-11-02 20:10:23.46712425 +0000 UTC m=+1321.983896839" Nov 02 20:10:23.500676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:23.500641 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:10:23.500847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:23.500691 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 20:10:24.372172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:24.372131 5800 generic.go:356] "Generic (PLEG): container finished" podID="ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" containerID="5a630d85374ffd2f0dc4253cd150601cde71e592cf2377820b1916bbd8936c28" exitCode=0 Nov 02 20:10:24.372685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:24.372192 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" event={"ID":"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e","Type":"ContainerDied","Data":"5a630d85374ffd2f0dc4253cd150601cde71e592cf2377820b1916bbd8936c28"} Nov 02 20:10:24.499266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:24.499234 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:10:24.499465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:24.499315 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 20:10:25.499703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.499667 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:10:25.500257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.500222 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 20:10:25.804546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.804504 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:25.829915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.824551 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-kube-api-access-7gllv\" (UniqueName: \"kubernetes.io/projected/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-kube-api-access-7gllv\") pod \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " Nov 02 20:10:25.829915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.824619 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-openshift-ca\") pod \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " Nov 02 20:10:25.829915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.824658 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-xks-ca\") pod \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\" (UID: \"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e\") " Nov 02 20:10:25.846705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.831633 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-openshift-ca" pod "ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" (UID: "ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:10:25.869504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.864447 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-kube-api-access-7gllv" pod "ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" (UID: "ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e"). InnerVolumeSpecName "kube-api-access-7gllv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:10:25.873301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.872402 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-xks-ca" pod "ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" (UID: "ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:10:25.934306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.933369 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gllv\" (UniqueName: \"kubernetes.io/projected/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-kube-api-access-7gllv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:25.934306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.933401 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:25.934306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:25.933418 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:10:26.470343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:26.462243 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" event={"ID":"ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e","Type":"ContainerDied","Data":"3f6084912c6e6e5d39ac0669cbce121f9c55d7f09df78b99452ba9c3f3be9b98"} Nov 02 20:10:26.470343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:26.462358 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6084912c6e6e5d39ac0669cbce121f9c55d7f09df78b99452ba9c3f3be9b98" Nov 02 20:10:26.470343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:26.462447 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j" Nov 02 20:10:26.509306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:26.502356 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:10:26.509306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:26.502413 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 20:10:27.521305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:27.513626 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:10:27.521305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:27.513689 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 20:10:27.620308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:27.619497 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:27.669305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:27.664417 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:27.740306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:27.724038 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:28.499510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.499471 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 02 20:10:28.499789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.499551 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 02 20:10:28.590680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.590645 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4"] Nov 02 20:10:28.591246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.591233 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" containerName="revoke-keys" Nov 02 20:10:28.591310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.591247 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" containerName="revoke-keys" Nov 02 20:10:28.591425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.591414 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" containerName="revoke-keys" Nov 02 20:10:28.595134 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.595115 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.644625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.644592 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 20:10:28.644924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.644895 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 20:10:28.648581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.648558 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 20:10:28.648718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.648656 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rdqzr\"" Nov 02 20:10:28.653358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.653334 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4"] Nov 02 20:10:28.691209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.691157 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgln\" (UniqueName: \"kubernetes.io/projected/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-kube-api-access-clgln\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.692112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.691217 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.692112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.691325 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.692112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.691478 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.692112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.691539 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.692112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.691584 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.692112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.691655 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.792601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.792504 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.792601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.792582 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.792853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.792675 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgln\" (UniqueName: \"kubernetes.io/projected/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-kube-api-access-clgln\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.792853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.792708 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.792853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.792758 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.793035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.792893 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.793035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.792952 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.793827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.793794 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.797478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.797448 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.797607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.797566 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.797672 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.797629 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.798017 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.797990 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.846165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.846127 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.846668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.846644 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgln\" (UniqueName: \"kubernetes.io/projected/d0bf9a44-5ba5-442c-b9c3-443ff0678c9e-kube-api-access-clgln\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h8bs4\" (UID: \"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:28.914716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:28.914680 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:29.368138 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:10:29.368099 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bf9a44_5ba5_442c_b9c3_443ff0678c9e.slice/crio-6342527fc285237e67e1808be6c1c67da83c44e2624dcff5eae032208f497a38 WatchSource:0}: Error finding container 6342527fc285237e67e1808be6c1c67da83c44e2624dcff5eae032208f497a38: Status 404 returned error can't find the container with id 6342527fc285237e67e1808be6c1c67da83c44e2624dcff5eae032208f497a38 Nov 02 20:10:29.368435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:29.368413 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4"] Nov 02 20:10:29.370444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:29.370428 5800 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:10:29.523637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:29.523593 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" event={"ID":"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e","Type":"ContainerStarted","Data":"6342527fc285237e67e1808be6c1c67da83c44e2624dcff5eae032208f497a38"} Nov 02 20:10:29.983558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:29.983524 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:29.984048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:29.983822 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:29.985032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:29.985010 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:10:33.379412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:33.379379 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:33.379724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:33.379443 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:33.562961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:33.562921 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" event={"ID":"d0bf9a44-5ba5-442c-b9c3-443ff0678c9e","Type":"ContainerStarted","Data":"8efd69418fd20ea040435a5abb2380ee0db867bd96a0f45735e6d3186bf3b386"} Nov 02 20:10:33.563180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:33.563017 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:34.637515 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:34.637473 5800 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-h8bs4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:10:34.637908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:34.637558 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" podUID="d0bf9a44-5ba5-442c-b9c3-443ff0678c9e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:10:37.611159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:37.611126 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" Nov 02 20:10:37.815492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:37.815436 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h8bs4" podStartSLOduration=5.807779869 podStartE2EDuration="9.81542021s" totalImagesPullingTime="4.007640341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:10:33.764645035 +0000 UTC m=+1332.281417620" watchObservedRunningTime="2026-11-02 20:10:37.81542021 +0000 UTC m=+1336.332192794" Nov 02 20:10:48.626534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:48.626499 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:10:48.627012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:48.626559 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:10:50.747136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:50.747104 5800 generic.go:356] "Generic (PLEG): container finished" podID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerID="99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685" exitCode=0 Nov 02 20:10:50.747697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:50.747178 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" event={"ID":"bb9856b5-90d2-4e34-84a7-43724bc9e1cd","Type":"ContainerDied","Data":"99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685"} Nov 02 20:10:50.747697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:50.747627 5800 scope.go:122] "RemoveContainer" containerID="99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685" Nov 02 20:10:50.749272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:50.749240 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:50.749398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:50.749343 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:50.749398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:50.749386 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:10:51.761736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:51.761692 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" event={"ID":"bb9856b5-90d2-4e34-84a7-43724bc9e1cd","Type":"ContainerStarted","Data":"e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca"} Nov 02 20:10:52.697044 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:10:52.696978 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:10:52.730253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:52.729810 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:10:52.730253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:52.729925 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 20:10:52.730253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:52.729969 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 20:10:53.747317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:53.746837 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 20:10:53.747317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:53.746901 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 20:10:54.735111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:54.735078 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 20:10:54.735440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:54.735409 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 20:10:55.732304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:55.731540 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 20:10:55.732304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:55.731604 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 20:10:56.732664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:56.732622 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 20:10:56.733080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:56.732724 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 20:10:57.735323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:57.735274 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:10:57.735774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:57.735691 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:10:57.736506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:10:57.736483 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:11:22.169260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:22.169176 5800 scope.go:122] "RemoveContainer" containerID="0a2dfc3cf4ed9d5dcfbb47fcb5a3a7851f3a2977e59e6acb45d3c9d27fa3f7ab" Nov 02 20:11:22.179327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:22.179275 5800 scope.go:122] "RemoveContainer" containerID="cee557e123bba05ef03eeb828f610950af5068a1b600bd6ad6214454a922b0c8" Nov 02 20:11:22.188386 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:22.188369 5800 scope.go:122] "RemoveContainer" containerID="5dc8bdb24f6ff6f8ea6212d0caeb5a75a092d8baddeab0b8ab93068f9a5c46b3" Nov 02 20:11:22.699120 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:11:22.699083 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:11:52.697071 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:11:52.697032 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:11:56.932443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:56.932407 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g"] Nov 02 20:11:56.938042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:56.938012 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:56.987884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:56.987850 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-671e7c-dockercfg-zpltr\"" Nov 02 20:11:56.990165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:56.990123 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g"] Nov 02 20:11:57.029481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.029447 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-671e7c\"" Nov 02 20:11:57.041564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.041517 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwq4p\" (UniqueName: \"kubernetes.io/projected/5475d899-987e-4fd5-a697-5208740a47f4-kube-api-access-nwq4p\") pod \"payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:57.041777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.041714 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5475d899-987e-4fd5-a697-5208740a47f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:57.100887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.100855 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6"] Nov 02 20:11:57.105603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.105579 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.143770 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.143736 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwq4p\" (UniqueName: \"kubernetes.io/projected/5475d899-987e-4fd5-a697-5208740a47f4-kube-api-access-nwq4p\") pod \"payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:57.143962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.143853 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5475d899-987e-4fd5-a697-5208740a47f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:57.144712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.144686 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5475d899-987e-4fd5-a697-5208740a47f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:57.158024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.157994 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6"] Nov 02 20:11:57.245802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.236102 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwq4p\" (UniqueName: \"kubernetes.io/projected/5475d899-987e-4fd5-a697-5208740a47f4-kube-api-access-nwq4p\") pod \"payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:57.245802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.245575 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.245802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.245737 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgxt\" (UniqueName: \"kubernetes.io/projected/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-kube-api-access-hcgxt\") pod \"payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.255371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.255340 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:57.351320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.348092 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.351320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.348268 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgxt\" (UniqueName: \"kubernetes.io/projected/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-kube-api-access-hcgxt\") pod \"payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.351320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.349738 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.402141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.401994 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgxt\" (UniqueName: \"kubernetes.io/projected/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-kube-api-access-hcgxt\") pod \"payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.420512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.420482 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:57.646230 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.646201 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g"] Nov 02 20:11:57.646523 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:11:57.646489 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5475d899_987e_4fd5_a697_5208740a47f4.slice/crio-474784081e9b8d4ddb9b049698ef4b10d21bcb3eb2a8f86196242715bda2e6fc WatchSource:0}: Error finding container 474784081e9b8d4ddb9b049698ef4b10d21bcb3eb2a8f86196242715bda2e6fc: Status 404 returned error can't find the container with id 474784081e9b8d4ddb9b049698ef4b10d21bcb3eb2a8f86196242715bda2e6fc Nov 02 20:11:57.811031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:57.810992 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6"] Nov 02 20:11:58.378694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.378653 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" event={"ID":"5475d899-987e-4fd5-a697-5208740a47f4","Type":"ContainerStarted","Data":"e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f"} Nov 02 20:11:58.379169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.378720 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" event={"ID":"5475d899-987e-4fd5-a697-5208740a47f4","Type":"ContainerStarted","Data":"474784081e9b8d4ddb9b049698ef4b10d21bcb3eb2a8f86196242715bda2e6fc"} Nov 02 20:11:58.379169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.378846 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:11:58.380544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.380519 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" event={"ID":"78ce45cb-959d-4e11-bcb5-8012f7f51ee3","Type":"ContainerStarted","Data":"952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82"} Nov 02 20:11:58.380673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.380561 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" event={"ID":"78ce45cb-959d-4e11-bcb5-8012f7f51ee3","Type":"ContainerStarted","Data":"88c36ae40822b6f2484d269bfdc8a810af69fe1c0f43e242a3eb58e56d4ee4ab"} Nov 02 20:11:58.380673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.380601 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:11:58.489791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.489737 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" podStartSLOduration=2.489721216 podStartE2EDuration="2.489721216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:58.483428911 +0000 UTC m=+1417.000201498" watchObservedRunningTime="2026-11-02 20:11:58.489721216 +0000 UTC m=+1417.006493801" Nov 02 20:11:58.592235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:11:58.591990 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" podStartSLOduration=1.591970975 podStartE2EDuration="1.591970975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:11:58.588683266 +0000 UTC m=+1417.105455852" watchObservedRunningTime="2026-11-02 20:11:58.591970975 +0000 UTC m=+1417.108743561" Nov 02 20:12:09.391973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:09.391942 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:12:09.392375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:09.392029 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:12:22.697586 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:12:22.697548 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:12:25.017359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.005883 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq"] Nov 02 20:12:25.017359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.011626 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.060302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.060063 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq"] Nov 02 20:12:25.093314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.092457 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.093314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.092561 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9xk\" (UniqueName: \"kubernetes.io/projected/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-kube-api-access-gq9xk\") pod \"payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.100850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.099664 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-341fb2\"" Nov 02 20:12:25.100850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.100320 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-341fb2-dockercfg-9xrfc\"" Nov 02 20:12:25.182304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.175836 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf"] Nov 02 20:12:25.189518 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.189488 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.196416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.194267 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.196416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.194331 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9xk\" (UniqueName: \"kubernetes.io/projected/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-kube-api-access-gq9xk\") pod \"payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.196416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.194398 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcm7r\" (UniqueName: \"kubernetes.io/projected/5495dc82-30ff-4357-94c8-3a077626f070-kube-api-access-kcm7r\") pod \"payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.196416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.194489 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5495dc82-30ff-4357-94c8-3a077626f070-plugins-config-volume\") pod \"payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.196416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.195394 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.245006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.244961 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf"] Nov 02 20:12:25.271133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.271017 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9xk\" (UniqueName: \"kubernetes.io/projected/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-kube-api-access-gq9xk\") pod \"payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.295467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.295431 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5495dc82-30ff-4357-94c8-3a077626f070-plugins-config-volume\") pod \"payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.295666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.295558 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcm7r\" (UniqueName: \"kubernetes.io/projected/5495dc82-30ff-4357-94c8-3a077626f070-kube-api-access-kcm7r\") pod \"payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.296776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.296704 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5495dc82-30ff-4357-94c8-3a077626f070-plugins-config-volume\") pod \"payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.332305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.332229 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:25.349535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.349507 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcm7r\" (UniqueName: \"kubernetes.io/projected/5495dc82-30ff-4357-94c8-3a077626f070-kube-api-access-kcm7r\") pod \"payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.525738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.525635 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:25.719520 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:12:25.719490 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed9aa4d_8cfc_4105_b17e_3f0a85d011a4.slice/crio-cc6e0cf849ecea9df91f02899ba3ac954390a140b4660fa284c90833f5bc313d WatchSource:0}: Error finding container cc6e0cf849ecea9df91f02899ba3ac954390a140b4660fa284c90833f5bc313d: Status 404 returned error can't find the container with id cc6e0cf849ecea9df91f02899ba3ac954390a140b4660fa284c90833f5bc313d Nov 02 20:12:25.720118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.720096 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq"] Nov 02 20:12:25.896331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:25.896302 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf"] Nov 02 20:12:25.896609 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:12:25.896587 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5495dc82_30ff_4357_94c8_3a077626f070.slice/crio-5e9859a1bc47669bc60b228f57bd188d0a2de4a4487c5552fe9a5f658db4a327 WatchSource:0}: Error finding container 5e9859a1bc47669bc60b228f57bd188d0a2de4a4487c5552fe9a5f658db4a327: Status 404 returned error can't find the container with id 5e9859a1bc47669bc60b228f57bd188d0a2de4a4487c5552fe9a5f658db4a327 Nov 02 20:12:26.615701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.615661 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" event={"ID":"5495dc82-30ff-4357-94c8-3a077626f070","Type":"ContainerStarted","Data":"913200d1f3193c51313405db8fb73a81ea9269ac671e9e5206604edb15f1272e"} Nov 02 20:12:26.616179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.615728 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:26.616179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.615756 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" event={"ID":"5495dc82-30ff-4357-94c8-3a077626f070","Type":"ContainerStarted","Data":"5e9859a1bc47669bc60b228f57bd188d0a2de4a4487c5552fe9a5f658db4a327"} Nov 02 20:12:26.617348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.617318 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" event={"ID":"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4","Type":"ContainerStarted","Data":"01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604"} Nov 02 20:12:26.617481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.617362 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" event={"ID":"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4","Type":"ContainerStarted","Data":"cc6e0cf849ecea9df91f02899ba3ac954390a140b4660fa284c90833f5bc313d"} Nov 02 20:12:26.617481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.617415 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:26.723453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.723402 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" podStartSLOduration=1.723389102 podStartE2EDuration="1.723389102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:26.720792566 +0000 UTC m=+1445.237565152" watchObservedRunningTime="2026-11-02 20:12:26.723389102 +0000 UTC m=+1445.240161687" Nov 02 20:12:26.830298 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:26.830239 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" podStartSLOduration=2.830217959 podStartE2EDuration="2.830217959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:26.826864616 +0000 UTC m=+1445.343637202" watchObservedRunningTime="2026-11-02 20:12:26.830217959 +0000 UTC m=+1445.346990545" Nov 02 20:12:31.169432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.169331 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf"] Nov 02 20:12:31.175466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.175444 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.224632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.224599 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a3021d-openshift-default-dockercfg-svzdx\"" Nov 02 20:12:31.275672 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.275641 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf"] Nov 02 20:12:31.281221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281189 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-certs\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.281812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281236 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5wf\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-kube-api-access-bn5wf\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.281812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281372 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-data\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.281812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281472 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-token\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.281812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281548 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-podinfo\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.281812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281571 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istiod-ca-cert\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.281812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281610 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-credential-socket\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.282352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281806 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-socket\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.282352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.281851 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-envoy\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.383682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383646 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-podinfo\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.383682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383684 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istiod-ca-cert\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.383893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383726 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-credential-socket\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.383893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383780 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-socket\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.383893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383804 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-envoy\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.383893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383853 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-certs\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.383893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383891 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5wf\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-kube-api-access-bn5wf\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.384051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383930 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-data\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.384051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.383992 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-token\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.384693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.384664 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-credential-socket\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.384837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.384762 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-socket\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.384909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.384874 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-certs\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.384969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.384911 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istiod-ca-cert\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.385023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.384963 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-data\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.387074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.387052 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-envoy\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.387541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.387513 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-podinfo\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.441926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.441846 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5wf\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-kube-api-access-bn5wf\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.442102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.441991 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-token\") pod \"e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.494337 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.494301 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:31.930478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.930449 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf"] Nov 02 20:12:31.931258 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:12:31.931230 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8782ae10_2e36_4da3_b1d3_cabb1ad325fa.slice/crio-f11823ca05a81ccc0ec9dd4ec9af1af4284159cea838f0c6a0474d987869d4a8 WatchSource:0}: Error finding container f11823ca05a81ccc0ec9dd4ec9af1af4284159cea838f0c6a0474d987869d4a8: Status 404 returned error can't find the container with id f11823ca05a81ccc0ec9dd4ec9af1af4284159cea838f0c6a0474d987869d4a8 Nov 02 20:12:31.934011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.933974 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:31.934101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.934066 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:31.934141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:31.934111 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:32.675100 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:32.675058 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" event={"ID":"8782ae10-2e36-4da3-b1d3-cabb1ad325fa","Type":"ContainerStarted","Data":"a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916"} Nov 02 20:12:32.675642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:32.675125 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" event={"ID":"8782ae10-2e36-4da3-b1d3-cabb1ad325fa","Type":"ContainerStarted","Data":"f11823ca05a81ccc0ec9dd4ec9af1af4284159cea838f0c6a0474d987869d4a8"} Nov 02 20:12:32.867640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:32.867587 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podStartSLOduration=1.86757153 podStartE2EDuration="1.86757153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:32.86452481 +0000 UTC m=+1451.381297397" watchObservedRunningTime="2026-11-02 20:12:32.86757153 +0000 UTC m=+1451.384344115" Nov 02 20:12:33.495062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:33.495021 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:33.500850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:33.500814 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:33.688016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:33.687986 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:33.689155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:33.689134 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:12:37.633490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:37.633458 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:12:37.633903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:37.633537 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:12:39.234275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.234238 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn"] Nov 02 20:12:39.238825 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.238800 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:39.290780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.290749 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn"] Nov 02 20:12:39.332726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.332689 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a3021d\"" Nov 02 20:12:39.332925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.332820 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a3021d-dockercfg-gbzn8\"" Nov 02 20:12:39.435041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.435001 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0971513-3f2f-49b1-814b-23362b80e9d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:39.435614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.435567 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kss\" (UniqueName: \"kubernetes.io/projected/f0971513-3f2f-49b1-814b-23362b80e9d0-kube-api-access-n7kss\") pod \"payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:39.536904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.536851 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0971513-3f2f-49b1-814b-23362b80e9d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:39.537103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.537053 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kss\" (UniqueName: \"kubernetes.io/projected/f0971513-3f2f-49b1-814b-23362b80e9d0-kube-api-access-n7kss\") pod \"payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:39.537647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.537622 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0971513-3f2f-49b1-814b-23362b80e9d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:39.594473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.594344 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kss\" (UniqueName: \"kubernetes.io/projected/f0971513-3f2f-49b1-814b-23362b80e9d0-kube-api-access-n7kss\") pod \"payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:39.863838 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:39.863743 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:40.233200 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:12:40.233170 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0971513_3f2f_49b1_814b_23362b80e9d0.slice/crio-8de4033199e3603493660a559073760e911074a810205627c0c95934465caa39 WatchSource:0}: Error finding container 8de4033199e3603493660a559073760e911074a810205627c0c95934465caa39: Status 404 returned error can't find the container with id 8de4033199e3603493660a559073760e911074a810205627c0c95934465caa39 Nov 02 20:12:40.234075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:40.234040 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn"] Nov 02 20:12:40.755336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:40.753857 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" event={"ID":"f0971513-3f2f-49b1-814b-23362b80e9d0","Type":"ContainerStarted","Data":"41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d"} Nov 02 20:12:40.755336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:40.753924 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" event={"ID":"f0971513-3f2f-49b1-814b-23362b80e9d0","Type":"ContainerStarted","Data":"8de4033199e3603493660a559073760e911074a810205627c0c95934465caa39"} Nov 02 20:12:40.864602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:40.864549 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" podStartSLOduration=1.864535772 podStartE2EDuration="1.864535772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:40.862309989 +0000 UTC m=+1459.379082574" watchObservedRunningTime="2026-11-02 20:12:40.864535772 +0000 UTC m=+1459.381308357" Nov 02 20:12:41.761959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:41.761922 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:52.697138 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:12:52.697102 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:12:52.773468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:52.773436 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:12:54.128623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.128583 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj"] Nov 02 20:12:54.133501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.133479 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149313 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4591daec-4b8a-4047-b2d7-1e1826365b7e-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149367 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149560 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149586 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149678 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149738 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149789 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149814 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.149915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.149872 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmhs\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-kube-api-access-zrmhs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.182451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.182420 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a4ab0-openshift-default-dockercfg-xzj7t\"" Nov 02 20:12:54.236048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.236014 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj"] Nov 02 20:12:54.255974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.255932 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.256713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.256571 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.256849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.256686 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257027 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257082 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257119 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmhs\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-kube-api-access-zrmhs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257145 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4591daec-4b8a-4047-b2d7-1e1826365b7e-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257201 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257230 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257371 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257405 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.257789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.257767 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.258935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.258891 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4591daec-4b8a-4047-b2d7-1e1826365b7e-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.259637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.259612 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.260759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.260713 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.261382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.261355 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.312215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.312188 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.312521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.312501 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmhs\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-kube-api-access-zrmhs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.449309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.449208 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:54.927089 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:12:54.927053 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4591daec_4b8a_4047_b2d7_1e1826365b7e.slice/crio-4641ed8669d180b6db2dfb194a60e59e539de9bde93133de417e06a197e85a0a WatchSource:0}: Error finding container 4641ed8669d180b6db2dfb194a60e59e539de9bde93133de417e06a197e85a0a: Status 404 returned error can't find the container with id 4641ed8669d180b6db2dfb194a60e59e539de9bde93133de417e06a197e85a0a Nov 02 20:12:54.927247 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.927166 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj"] Nov 02 20:12:54.930325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.930269 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:54.930443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.930360 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:54.930443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:54.930388 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:12:55.901420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:55.901372 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" event={"ID":"4591daec-4b8a-4047-b2d7-1e1826365b7e","Type":"ContainerStarted","Data":"b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a"} Nov 02 20:12:55.901822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:55.901448 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" event={"ID":"4591daec-4b8a-4047-b2d7-1e1826365b7e","Type":"ContainerStarted","Data":"4641ed8669d180b6db2dfb194a60e59e539de9bde93133de417e06a197e85a0a"} Nov 02 20:12:56.463338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:56.454668 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:12:56.463338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:56.459378 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 20:12:56.463338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:56.459448 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 20:12:57.456335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:57.455155 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 20:12:57.456335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:57.455219 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 20:12:58.453864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:58.453824 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 20:12:58.454211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:58.454175 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 20:12:59.450304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:59.449666 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 20:12:59.450304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:12:59.449753 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 20:13:00.450305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:00.449666 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 20:13:00.450305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:00.449734 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 20:13:01.449788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:01.449749 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 02 20:13:01.449999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:01.449818 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 02 20:13:03.490493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:03.490442 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:13:03.491211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:03.490522 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:13:03.883894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:03.883861 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:13:03.884379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:03.884347 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:13:03.888095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:03.888068 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:13:04.094505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.094456 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podStartSLOduration=10.094441497 podStartE2EDuration="10.094441497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:12:56.019598073 +0000 UTC m=+1474.536370659" watchObservedRunningTime="2026-11-02 20:13:04.094441497 +0000 UTC m=+1482.611214081" Nov 02 20:13:04.108822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.108777 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c"] Nov 02 20:13:04.114508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.114491 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.172715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.172624 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-data\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173070 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173232 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-certs\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173498 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-token\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173547 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggw9\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-kube-api-access-mggw9\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173578 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-podinfo\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173620 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693f9eea-24e2-446d-b5e9-cf8098787d4a-istiod-ca-cert\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173664 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-envoy\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.173820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.173696 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-credential-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.175398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.175375 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c"] Nov 02 20:13:04.194725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.194692 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-671e7c-openshift-default-dockercfg-ctrxx\"" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276220 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-data\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276524 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276592 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-certs\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276648 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-token\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276723 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggw9\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-kube-api-access-mggw9\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276754 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-podinfo\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276800 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693f9eea-24e2-446d-b5e9-cf8098787d4a-istiod-ca-cert\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276849 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-envoy\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.276882 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-credential-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.277460 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-credential-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.277721 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-data\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.277982 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-socket\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.278224 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-certs\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.281385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.279857 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693f9eea-24e2-446d-b5e9-cf8098787d4a-istiod-ca-cert\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.282364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.282087 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-podinfo\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.282364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.282157 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-envoy\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.332374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.332341 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-token\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.332638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.332614 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggw9\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-kube-api-access-mggw9\") pod \"e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.430463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.430369 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:04.855962 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:13:04.855930 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693f9eea_24e2_446d_b5e9_cf8098787d4a.slice/crio-99857c6e1590d562d3229f61403aba45ad77743d09eb5ea09c77b9093adcc1a9 WatchSource:0}: Error finding container 99857c6e1590d562d3229f61403aba45ad77743d09eb5ea09c77b9093adcc1a9: Status 404 returned error can't find the container with id 99857c6e1590d562d3229f61403aba45ad77743d09eb5ea09c77b9093adcc1a9 Nov 02 20:13:04.858789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.858755 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:04.858881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.858783 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c"] Nov 02 20:13:04.858881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.858836 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:04.858881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:04.858864 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:13:05.010426 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.010389 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" event={"ID":"693f9eea-24e2-446d-b5e9-cf8098787d4a","Type":"ContainerStarted","Data":"bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f"} Nov 02 20:13:05.010613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.010463 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" event={"ID":"693f9eea-24e2-446d-b5e9-cf8098787d4a","Type":"ContainerStarted","Data":"99857c6e1590d562d3229f61403aba45ad77743d09eb5ea09c77b9093adcc1a9"} Nov 02 20:13:05.056212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.056171 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq"] Nov 02 20:13:05.056427 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.056238 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9bbf8e8-748zq"] Nov 02 20:13:05.149964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.149852 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podStartSLOduration=1.149836945 podStartE2EDuration="1.149836945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:05.146851242 +0000 UTC m=+1483.663623828" watchObservedRunningTime="2026-11-02 20:13:05.149836945 +0000 UTC m=+1483.666609530" Nov 02 20:13:05.431054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.430959 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:05.432743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.432713 5800 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:13:05.432832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.432780 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 20:13:05.623507 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:05.623467 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0164d53-b5f8-4acc-b097-5a40a1ca4f71" path="/var/lib/kubelet/pods/f0164d53-b5f8-4acc-b097-5a40a1ca4f71/volumes" Nov 02 20:13:06.438729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:06.438690 5800 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:13:06.440234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:06.440198 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 20:13:07.440306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:07.436649 5800 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:13:07.440306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:07.436712 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 20:13:08.441307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:08.436714 5800 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:13:08.441307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:08.436783 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 20:13:09.431640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:09.431602 5800 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:13:09.433623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:09.433586 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 20:13:10.441757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:10.441715 5800 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 02 20:13:10.442414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:10.442380 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 02 20:13:11.435975 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:11.435944 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:11.436250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:11.436230 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:11.437038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:11.437022 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:13:21.893858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:21.893832 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:13:21.897918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:21.897894 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:13:22.045553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:22.045515 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr"] Nov 02 20:13:22.091848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:22.091814 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41ad198a-xspfr"] Nov 02 20:13:22.697554 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:13:22.697515 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:13:23.623930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:23.623889 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8943ff71-b8e8-437c-ab45-7bb85c9fd6a9" path="/var/lib/kubelet/pods/8943ff71-b8e8-437c-ab45-7bb85c9fd6a9/volumes" Nov 02 20:13:25.044851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:25.044817 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr"] Nov 02 20:13:25.091384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:25.091353 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3300fffd-qz9rr"] Nov 02 20:13:25.673656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:25.673620 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f601a7-f7aa-4c7f-a0cc-029025f8db90" path="/var/lib/kubelet/pods/e9f601a7-f7aa-4c7f-a0cc-029025f8db90/volumes" Nov 02 20:13:36.045051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:36.045012 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc"] Nov 02 20:13:36.091559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:36.091526 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d80ba10d-7j7nc"] Nov 02 20:13:37.623321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:37.623267 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf827857-c1ef-445a-9001-f875c8751836" path="/var/lib/kubelet/pods/bf827857-c1ef-445a-9001-f875c8751836/volumes" Nov 02 20:13:52.713557 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:13:52.713515 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:13:53.227476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.227401 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld"] Nov 02 20:13:53.244507 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.244357 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c97fd-xlqld"] Nov 02 20:13:53.255175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.255145 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27"] Nov 02 20:13:53.290719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.290679 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27"] Nov 02 20:13:53.291172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.291153 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:53.390342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.359674 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-15203f-dockercfg-sq676\"" Nov 02 20:13:53.390342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.359841 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-15203f\"" Nov 02 20:13:53.502449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.502300 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgml\" (UniqueName: \"kubernetes.io/projected/3ead968e-a8c6-4d69-8856-1b932ee3e62c-kube-api-access-cfgml\") pod \"payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:53.502785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.502761 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead968e-a8c6-4d69-8856-1b932ee3e62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:53.603894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.603862 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgml\" (UniqueName: \"kubernetes.io/projected/3ead968e-a8c6-4d69-8856-1b932ee3e62c-kube-api-access-cfgml\") pod \"payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:53.604104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.603909 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead968e-a8c6-4d69-8856-1b932ee3e62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:53.604758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.604738 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead968e-a8c6-4d69-8856-1b932ee3e62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:53.624623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.624591 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abd0b11-b69b-4eaf-ab81-2109705adf0a" path="/var/lib/kubelet/pods/9abd0b11-b69b-4eaf-ab81-2109705adf0a/volumes" Nov 02 20:13:53.668330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.668018 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgml\" (UniqueName: \"kubernetes.io/projected/3ead968e-a8c6-4d69-8856-1b932ee3e62c-kube-api-access-cfgml\") pod \"payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:53.916376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:53.916275 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:54.083808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:54.083640 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk"] Nov 02 20:13:54.352035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:54.351997 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27"] Nov 02 20:13:54.562372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:54.562333 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" event={"ID":"3ead968e-a8c6-4d69-8856-1b932ee3e62c","Type":"ContainerStarted","Data":"dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed"} Nov 02 20:13:54.562650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:54.562630 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:13:54.562805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:54.562786 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" event={"ID":"3ead968e-a8c6-4d69-8856-1b932ee3e62c","Type":"ContainerStarted","Data":"4a8f0a766a04bf9ad8a8326226ca725cc83228c537e83d614e8ef3073e443a67"} Nov 02 20:13:59.084078 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:13:59.084005 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" podUID="8b719543-4c07-4e26-ab54-a1e1483deb2e" containerName="payload-pre-processing" containerID="cri-o://1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175" gracePeriod=25 Nov 02 20:14:00.511101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.511006 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" podStartSLOduration=7.510991512 podStartE2EDuration="7.510991512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:13:54.673039919 +0000 UTC m=+1533.189812528" watchObservedRunningTime="2026-11-02 20:14:00.510991512 +0000 UTC m=+1539.027764106" Nov 02 20:14:00.512490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.512467 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk"] Nov 02 20:14:00.517178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.517161 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:00.570095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.570047 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk"] Nov 02 20:14:00.610637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.610599 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e0aa5c\"" Nov 02 20:14:00.610833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.610812 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e0aa5c-dockercfg-5qrw6\"" Nov 02 20:14:00.659636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.654711 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wf6r\" (UniqueName: \"kubernetes.io/projected/8de5a762-1a07-4969-9c1b-62c23463d813-kube-api-access-7wf6r\") pod \"payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:00.699180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.699150 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5a762-1a07-4969-9c1b-62c23463d813-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:00.758540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.758505 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd"] Nov 02 20:14:00.764666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.764582 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:00.768348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.768291 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd"] Nov 02 20:14:00.853304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.851213 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wf6r\" (UniqueName: \"kubernetes.io/projected/8de5a762-1a07-4969-9c1b-62c23463d813-kube-api-access-7wf6r\") pod \"payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:00.853304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.851481 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5a762-1a07-4969-9c1b-62c23463d813-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:00.853304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.852701 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5a762-1a07-4969-9c1b-62c23463d813-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:00.909995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.909966 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wf6r\" (UniqueName: \"kubernetes.io/projected/8de5a762-1a07-4969-9c1b-62c23463d813-kube-api-access-7wf6r\") pod \"payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:00.945697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.945666 5800 generic.go:356] "Generic (PLEG): container finished" podID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerID="a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916" exitCode=0 Nov 02 20:14:00.945823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.945734 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" event={"ID":"8782ae10-2e36-4da3-b1d3-cabb1ad325fa","Type":"ContainerDied","Data":"a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916"} Nov 02 20:14:00.946222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.946208 5800 scope.go:122] "RemoveContainer" containerID="a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916" Nov 02 20:14:00.948121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.948088 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:00.948235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.948194 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:00.948327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.948246 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:14:00.953140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.953117 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk9ds\" (UniqueName: \"kubernetes.io/projected/5400ca29-d118-435b-b432-67931f51e264-kube-api-access-rk9ds\") pod \"payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:00.953366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:00.953339 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400ca29-d118-435b-b432-67931f51e264-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:01.057068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.057023 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk9ds\" (UniqueName: \"kubernetes.io/projected/5400ca29-d118-435b-b432-67931f51e264-kube-api-access-rk9ds\") pod \"payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:01.057364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.057340 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400ca29-d118-435b-b432-67931f51e264-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:01.058835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.058806 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400ca29-d118-435b-b432-67931f51e264-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:01.060242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.060216 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb"] Nov 02 20:14:01.078437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.078405 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9db841-dclnb"] Nov 02 20:14:01.116248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.116211 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk9ds\" (UniqueName: \"kubernetes.io/projected/5400ca29-d118-435b-b432-67931f51e264-kube-api-access-rk9ds\") pod \"payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:01.134570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.134532 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:01.388352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.388258 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:01.547023 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:14:01.546993 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de5a762_1a07_4969_9c1b_62c23463d813.slice/crio-051e4a9f5e272d8613c85c32b726d3fa6f757db4cf27db5d670029d4f0b3b9ee WatchSource:0}: Error finding container 051e4a9f5e272d8613c85c32b726d3fa6f757db4cf27db5d670029d4f0b3b9ee: Status 404 returned error can't find the container with id 051e4a9f5e272d8613c85c32b726d3fa6f757db4cf27db5d670029d4f0b3b9ee Nov 02 20:14:01.550505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.550377 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk"] Nov 02 20:14:01.624767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.624740 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a7c678-6bd4-485d-960d-fde51c690d14" path="/var/lib/kubelet/pods/57a7c678-6bd4-485d-960d-fde51c690d14/volumes" Nov 02 20:14:01.773877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.773848 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd"] Nov 02 20:14:01.774823 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:14:01.774797 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5400ca29_d118_435b_b432_67931f51e264.slice/crio-e625201bc8e2b4847affe4a058e1ccc7e04a82f0db77e3f7a8cd242ea6270340 WatchSource:0}: Error finding container e625201bc8e2b4847affe4a058e1ccc7e04a82f0db77e3f7a8cd242ea6270340: Status 404 returned error can't find the container with id e625201bc8e2b4847affe4a058e1ccc7e04a82f0db77e3f7a8cd242ea6270340 Nov 02 20:14:01.959435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.959403 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" event={"ID":"5400ca29-d118-435b-b432-67931f51e264","Type":"ContainerStarted","Data":"35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7"} Nov 02 20:14:01.959643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.959459 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" event={"ID":"5400ca29-d118-435b-b432-67931f51e264","Type":"ContainerStarted","Data":"e625201bc8e2b4847affe4a058e1ccc7e04a82f0db77e3f7a8cd242ea6270340"} Nov 02 20:14:01.959643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.959491 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:01.962317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.962255 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" event={"ID":"8782ae10-2e36-4da3-b1d3-cabb1ad325fa","Type":"ContainerStarted","Data":"03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d"} Nov 02 20:14:01.964027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.964002 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" event={"ID":"8de5a762-1a07-4969-9c1b-62c23463d813","Type":"ContainerStarted","Data":"7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0"} Nov 02 20:14:01.964143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.964047 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" event={"ID":"8de5a762-1a07-4969-9c1b-62c23463d813","Type":"ContainerStarted","Data":"051e4a9f5e272d8613c85c32b726d3fa6f757db4cf27db5d670029d4f0b3b9ee"} Nov 02 20:14:01.964143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:01.964123 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:02.068562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:02.068437 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" podStartSLOduration=2.068422334 podStartE2EDuration="2.068422334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:02.06475924 +0000 UTC m=+1540.581531825" watchObservedRunningTime="2026-11-02 20:14:02.068422334 +0000 UTC m=+1540.585194961" Nov 02 20:14:02.176960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:02.176905 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" podStartSLOduration=2.176889242 podStartE2EDuration="2.176889242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:02.169553049 +0000 UTC m=+1540.686325636" watchObservedRunningTime="2026-11-02 20:14:02.176889242 +0000 UTC m=+1540.693661819" Nov 02 20:14:02.494619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:02.494526 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:14:02.495700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:02.495674 5800 patch_prober.go:29] interesting pod/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:14:02.495850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:02.495727 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 20:14:03.494965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:03.494925 5800 patch_prober.go:29] interesting pod/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:14:03.495376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:03.494986 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 20:14:03.688229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:03.688198 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:14:04.494856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:04.494822 5800 patch_prober.go:29] interesting pod/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:14:04.495037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:04.494876 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 20:14:04.657578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:04.657528 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" podUID="8b719543-4c07-4e26-ab54-a1e1483deb2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:14:05.495655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:05.495618 5800 patch_prober.go:29] interesting pod/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:14:05.496046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:05.495677 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 20:14:05.899907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:05.899876 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:14:06.495667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:06.495631 5800 patch_prober.go:29] interesting pod/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:14:06.496153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:06.495694 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 20:14:07.495400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:07.495364 5800 patch_prober.go:29] interesting pod/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 02 20:14:07.495586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:07.495419 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 02 20:14:08.500396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:08.500362 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:14:08.501566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:08.501544 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:14:09.045116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:09.045079 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4"] Nov 02 20:14:09.048910 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:09.048888 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e7d177-6jmh4"] Nov 02 20:14:09.626310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:09.626263 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61d2a0a-440b-43ad-b43c-f5fc4171c7aa" path="/var/lib/kubelet/pods/c61d2a0a-440b-43ad-b43c-f5fc4171c7aa/volumes" Nov 02 20:14:12.976556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:12.976518 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:14:12.976953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:12.976599 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:14:14.646743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:14.646695 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" podUID="8b719543-4c07-4e26-ab54-a1e1483deb2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:14:19.388222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:19.388186 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:14:19.486246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:19.486214 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b719543-4c07-4e26-ab54-a1e1483deb2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b719543-4c07-4e26-ab54-a1e1483deb2e-plugins-config-volume\") pod \"8b719543-4c07-4e26-ab54-a1e1483deb2e\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " Nov 02 20:14:19.486467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:19.486263 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b719543-4c07-4e26-ab54-a1e1483deb2e-kube-api-access-lzp8k\" (UniqueName: \"kubernetes.io/projected/8b719543-4c07-4e26-ab54-a1e1483deb2e-kube-api-access-lzp8k\") pod \"8b719543-4c07-4e26-ab54-a1e1483deb2e\" (UID: \"8b719543-4c07-4e26-ab54-a1e1483deb2e\") " Nov 02 20:14:19.486640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:19.486614 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b719543-4c07-4e26-ab54-a1e1483deb2e-plugins-config-volume" pod "8b719543-4c07-4e26-ab54-a1e1483deb2e" (UID: "8b719543-4c07-4e26-ab54-a1e1483deb2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:19.486937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:19.486921 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b719543-4c07-4e26-ab54-a1e1483deb2e-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:19.488430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:19.488407 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b719543-4c07-4e26-ab54-a1e1483deb2e-kube-api-access-lzp8k" pod "8b719543-4c07-4e26-ab54-a1e1483deb2e" (UID: "8b719543-4c07-4e26-ab54-a1e1483deb2e"). InnerVolumeSpecName "kube-api-access-lzp8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:19.589715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:19.589676 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzp8k\" (UniqueName: \"kubernetes.io/projected/8b719543-4c07-4e26-ab54-a1e1483deb2e-kube-api-access-lzp8k\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:20.045577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.045536 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq"] Nov 02 20:14:20.049410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.049382 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39157f67-vt5qq"] Nov 02 20:14:20.114465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.114429 5800 generic.go:356] "Generic (PLEG): container finished" podID="8b719543-4c07-4e26-ab54-a1e1483deb2e" containerID="1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175" exitCode=0 Nov 02 20:14:20.114636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.114477 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" Nov 02 20:14:20.114636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.114518 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" event={"ID":"8b719543-4c07-4e26-ab54-a1e1483deb2e","Type":"ContainerDied","Data":"1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175"} Nov 02 20:14:20.114636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.114574 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk" event={"ID":"8b719543-4c07-4e26-ab54-a1e1483deb2e","Type":"ContainerDied","Data":"e534d7686fc12870f28f6bd27616f1f93cf099ef7c2119b6c1e0fb2cb1e703f7"} Nov 02 20:14:20.114636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.114604 5800 scope.go:122] "RemoveContainer" containerID="1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175" Nov 02 20:14:20.125414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.125396 5800 scope.go:122] "RemoveContainer" containerID="1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175" Nov 02 20:14:20.125670 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:14:20.125650 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175\": container with ID starting with 1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175 not found: ID does not exist" containerID="1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175" Nov 02 20:14:20.125762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.125678 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175"} err="failed to get container status \"1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175\": rpc error: code = NotFound desc = could not find container \"1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175\": container with ID starting with 1e4d3cb27fb5018c775f4d4bdf795d1e70d922363e5898a406067f9a37645175 not found: ID does not exist" Nov 02 20:14:20.269849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.269810 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk"] Nov 02 20:14:20.276060 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:20.276033 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-67jlk"] Nov 02 20:14:21.639248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:21.639205 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b719543-4c07-4e26-ab54-a1e1483deb2e" path="/var/lib/kubelet/pods/8b719543-4c07-4e26-ab54-a1e1483deb2e/volumes" Nov 02 20:14:21.639959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:21.639936 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5005ac4-f3d0-4e2d-80fb-585c6fee08bb" path="/var/lib/kubelet/pods/c5005ac4-f3d0-4e2d-80fb-585c6fee08bb/volumes" Nov 02 20:14:22.263438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:22.263398 5800 scope.go:122] "RemoveContainer" containerID="675976b7900d2d466364e2f1f501e94e0f694ed032c883bcc85f7bd5bdbfe1b8" Nov 02 20:14:22.274592 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:22.274570 5800 scope.go:122] "RemoveContainer" containerID="dcff7e5dd4e89ca4a257fd90b0fb0dbdaf047986313378cdcd00472a1a679c6b" Nov 02 20:14:22.284866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:22.284847 5800 scope.go:122] "RemoveContainer" containerID="490a1925ee6462ae989a2aae2b8168ac5e784a56a70f451c2f1446424b5d1dac" Nov 02 20:14:22.697228 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:14:22.697134 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:14:24.045032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:24.044997 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht"] Nov 02 20:14:24.093522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:24.093472 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a098493b-c4wht"] Nov 02 20:14:25.626379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:25.626349 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3877ccbe-0724-4170-bb40-76a54c5affc4" path="/var/lib/kubelet/pods/3877ccbe-0724-4170-bb40-76a54c5affc4/volumes" Nov 02 20:14:29.246633 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.246590 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8"] Nov 02 20:14:29.248526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.248496 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8b719543-4c07-4e26-ab54-a1e1483deb2e" containerName="payload-pre-processing" Nov 02 20:14:29.248705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.248678 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b719543-4c07-4e26-ab54-a1e1483deb2e" containerName="payload-pre-processing" Nov 02 20:14:29.248925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.248908 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b719543-4c07-4e26-ab54-a1e1483deb2e" containerName="payload-pre-processing" Nov 02 20:14:29.255723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.255697 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.299240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.299203 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8"] Nov 02 20:14:29.340947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.340915 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:14:29.341152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.341087 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:14:29.344754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.344717 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfbw\" (UniqueName: \"kubernetes.io/projected/7ab4c922-3941-4913-ab35-a35022eea89f-kube-api-access-jqfbw\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.345006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.344986 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.345392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.345136 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.352347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.352148 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:14:29.352347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.352243 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:14:29.446915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.446876 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.447100 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.446999 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfbw\" (UniqueName: \"kubernetes.io/projected/7ab4c922-3941-4913-ab35-a35022eea89f-kube-api-access-jqfbw\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.447157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.447133 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.447811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.447788 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.449478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.449458 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.500705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.500673 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfbw\" (UniqueName: \"kubernetes.io/projected/7ab4c922-3941-4913-ab35-a35022eea89f-kube-api-access-jqfbw\") pod \"maas-api-revoke-sub-22cda722-vzhc8\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.574580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.574547 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:29.953223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:29.953191 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8"] Nov 02 20:14:29.953712 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:14:29.953679 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab4c922_3941_4913_ab35_a35022eea89f.slice/crio-87b78bcef54f8e2c53909e7c91462089f32de5d4e88be44b4cdf1229a51f5112 WatchSource:0}: Error finding container 87b78bcef54f8e2c53909e7c91462089f32de5d4e88be44b4cdf1229a51f5112: Status 404 returned error can't find the container with id 87b78bcef54f8e2c53909e7c91462089f32de5d4e88be44b4cdf1229a51f5112 Nov 02 20:14:30.216910 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:30.216803 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" event={"ID":"7ab4c922-3941-4913-ab35-a35022eea89f","Type":"ContainerStarted","Data":"87b78bcef54f8e2c53909e7c91462089f32de5d4e88be44b4cdf1229a51f5112"} Nov 02 20:14:31.227333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:31.227195 5800 generic.go:356] "Generic (PLEG): container finished" podID="7ab4c922-3941-4913-ab35-a35022eea89f" containerID="7de0c920c2b4ae2597167870bccdf2fcc0d43cc752a8a5b1fbf33db94600f01c" exitCode=0 Nov 02 20:14:31.227333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:31.227276 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" event={"ID":"7ab4c922-3941-4913-ab35-a35022eea89f","Type":"ContainerDied","Data":"7de0c920c2b4ae2597167870bccdf2fcc0d43cc752a8a5b1fbf33db94600f01c"} Nov 02 20:14:32.422736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.422709 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:32.504185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.504159 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab4c922-3941-4913-ab35-a35022eea89f-kube-api-access-jqfbw\" (UniqueName: \"kubernetes.io/projected/7ab4c922-3941-4913-ab35-a35022eea89f-kube-api-access-jqfbw\") pod \"7ab4c922-3941-4913-ab35-a35022eea89f\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " Nov 02 20:14:32.504367 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.504262 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-openshift-ca\") pod \"7ab4c922-3941-4913-ab35-a35022eea89f\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " Nov 02 20:14:32.504472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.504454 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-xks-ca\") pod \"7ab4c922-3941-4913-ab35-a35022eea89f\" (UID: \"7ab4c922-3941-4913-ab35-a35022eea89f\") " Nov 02 20:14:32.504660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.504637 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-openshift-ca" pod "7ab4c922-3941-4913-ab35-a35022eea89f" (UID: "7ab4c922-3941-4913-ab35-a35022eea89f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:14:32.505023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.505004 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:32.506552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.506506 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-xks-ca" pod "7ab4c922-3941-4913-ab35-a35022eea89f" (UID: "7ab4c922-3941-4913-ab35-a35022eea89f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:14:32.506552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.506536 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab4c922-3941-4913-ab35-a35022eea89f-kube-api-access-jqfbw" pod "7ab4c922-3941-4913-ab35-a35022eea89f" (UID: "7ab4c922-3941-4913-ab35-a35022eea89f"). InnerVolumeSpecName "kube-api-access-jqfbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:14:32.607322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.607267 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ab4c922-3941-4913-ab35-a35022eea89f-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:32.607322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:32.607321 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqfbw\" (UniqueName: \"kubernetes.io/projected/7ab4c922-3941-4913-ab35-a35022eea89f-kube-api-access-jqfbw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:14:33.249754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:33.249724 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" Nov 02 20:14:33.249969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:33.249755 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8" event={"ID":"7ab4c922-3941-4913-ab35-a35022eea89f","Type":"ContainerDied","Data":"87b78bcef54f8e2c53909e7c91462089f32de5d4e88be44b4cdf1229a51f5112"} Nov 02 20:14:33.249969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:33.249814 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b78bcef54f8e2c53909e7c91462089f32de5d4e88be44b4cdf1229a51f5112" Nov 02 20:14:34.046120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:34.046083 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29"] Nov 02 20:14:34.092454 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:34.092421 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b67a77e-95j29"] Nov 02 20:14:35.624092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:35.624058 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3548f1c-0a36-4fb2-9775-675eb998acf4" path="/var/lib/kubelet/pods/f3548f1c-0a36-4fb2-9775-675eb998acf4/volumes" Nov 02 20:14:43.045156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:43.045114 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck"] Nov 02 20:14:43.090903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:43.090864 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8cb5322-nzrck"] Nov 02 20:14:43.626719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:43.626678 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2004c9-8d63-457d-95ff-1be18e241777" path="/var/lib/kubelet/pods/8a2004c9-8d63-457d-95ff-1be18e241777/volumes" Nov 02 20:14:47.045186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:47.045149 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm"] Nov 02 20:14:47.091442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:47.091357 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1a97324-7lxfm"] Nov 02 20:14:47.624910 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:14:47.624873 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d466fb2-0664-47b2-a51c-f8b40f26e5b6" path="/var/lib/kubelet/pods/8d466fb2-0664-47b2-a51c-f8b40f26e5b6/volumes" Nov 02 20:14:52.698491 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:14:52.698452 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:15:00.160922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.160879 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692"] Nov 02 20:15:00.161687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.161666 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7ab4c922-3941-4913-ab35-a35022eea89f" containerName="revoke-keys" Nov 02 20:15:00.161687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.161688 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab4c922-3941-4913-ab35-a35022eea89f" containerName="revoke-keys" Nov 02 20:15:00.161823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.161809 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab4c922-3941-4913-ab35-a35022eea89f" containerName="revoke-keys" Nov 02 20:15:00.166588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.166564 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49"] Nov 02 20:15:00.166735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.166693 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.171240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.171215 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.177597 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.177576 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl"] Nov 02 20:15:00.181819 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.181797 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.205809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.205782 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn"] Nov 02 20:15:00.211095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.211075 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q"] Nov 02 20:15:00.211262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.211210 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.215591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.215567 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9"] Nov 02 20:15:00.215710 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.215674 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.219950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.219929 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:15:00.220074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.219951 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.220074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.219936 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:15:00.247647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.247625 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:15:00.254811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.254784 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692"] Nov 02 20:15:00.258833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.258806 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49"] Nov 02 20:15:00.259552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.259531 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46st6\" (UniqueName: \"kubernetes.io/projected/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-kube-api-access-46st6\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.259625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.259562 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697zq\" (UniqueName: \"kubernetes.io/projected/189e8ce6-3ae1-43e0-a637-0300c3f9deff-kube-api-access-697zq\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.259625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.259596 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwbb\" (UniqueName: \"kubernetes.io/projected/59041d79-d3f5-42d6-bf89-633f228cd747-kube-api-access-nhwbb\") pod \"maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.259815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.259796 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189e8ce6-3ae1-43e0-a637-0300c3f9deff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.259852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.259838 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59041d79-d3f5-42d6-bf89-633f228cd747-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.260033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.260015 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.305209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.305159 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl"] Nov 02 20:15:00.308946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.308919 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn"] Nov 02 20:15:00.357564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.357507 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9"] Nov 02 20:15:00.361897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.361851 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba91842-792f-45bc-831a-60dd4419cdcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.362068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.361973 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tf7\" (UniqueName: \"kubernetes.io/projected/1ba91842-792f-45bc-831a-60dd4419cdcd-kube-api-access-98tf7\") pod \"maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.362141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.362116 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.362214 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46st6\" (UniqueName: \"kubernetes.io/projected/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-kube-api-access-46st6\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.362591 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697zq\" (UniqueName: \"kubernetes.io/projected/189e8ce6-3ae1-43e0-a637-0300c3f9deff-kube-api-access-697zq\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.362648 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b6ef9-c443-4760-bf82-462f49e08d23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.362689 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwbb\" (UniqueName: \"kubernetes.io/projected/59041d79-d3f5-42d6-bf89-633f228cd747-kube-api-access-nhwbb\") pod \"maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.362973 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d920b9b6-8c49-4320-9ca8-fbeb37c75703-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.363016 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.363444 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86v5g\" (UniqueName: \"kubernetes.io/projected/606b6ef9-c443-4760-bf82-462f49e08d23-kube-api-access-86v5g\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.363499 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189e8ce6-3ae1-43e0-a637-0300c3f9deff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.363519 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69h4\" (UniqueName: \"kubernetes.io/projected/d920b9b6-8c49-4320-9ca8-fbeb37c75703-kube-api-access-q69h4\") pod \"maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.364109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.363548 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59041d79-d3f5-42d6-bf89-633f228cd747-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.364769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.364746 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59041d79-d3f5-42d6-bf89-633f228cd747-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.365339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.365313 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189e8ce6-3ae1-43e0-a637-0300c3f9deff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.366096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.366075 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q"] Nov 02 20:15:00.415827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.415742 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46st6\" (UniqueName: \"kubernetes.io/projected/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-kube-api-access-46st6\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.416635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.416605 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwbb\" (UniqueName: \"kubernetes.io/projected/59041d79-d3f5-42d6-bf89-633f228cd747-kube-api-access-nhwbb\") pod \"maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.416954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.416928 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-697zq\" (UniqueName: \"kubernetes.io/projected/189e8ce6-3ae1-43e0-a637-0300c3f9deff-kube-api-access-697zq\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.464522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.464483 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69h4\" (UniqueName: \"kubernetes.io/projected/d920b9b6-8c49-4320-9ca8-fbeb37c75703-kube-api-access-q69h4\") pod \"maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.464755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.464594 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba91842-792f-45bc-831a-60dd4419cdcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.464755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.464620 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tf7\" (UniqueName: \"kubernetes.io/projected/1ba91842-792f-45bc-831a-60dd4419cdcd-kube-api-access-98tf7\") pod \"maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.464897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.464803 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b6ef9-c443-4760-bf82-462f49e08d23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.465357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.465335 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba91842-792f-45bc-831a-60dd4419cdcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.465457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.465435 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b6ef9-c443-4760-bf82-462f49e08d23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.465580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.465471 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d920b9b6-8c49-4320-9ca8-fbeb37c75703-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.465580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.465533 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86v5g\" (UniqueName: \"kubernetes.io/projected/606b6ef9-c443-4760-bf82-462f49e08d23-kube-api-access-86v5g\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.465964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.465947 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d920b9b6-8c49-4320-9ca8-fbeb37c75703-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.484790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.484765 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:00.493448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.493418 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:00.501993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.501771 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:00.518554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.518525 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69h4\" (UniqueName: \"kubernetes.io/projected/d920b9b6-8c49-4320-9ca8-fbeb37c75703-kube-api-access-q69h4\") pod \"maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.518671 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.518652 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tf7\" (UniqueName: \"kubernetes.io/projected/1ba91842-792f-45bc-831a-60dd4419cdcd-kube-api-access-98tf7\") pod \"maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.519520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.519500 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86v5g\" (UniqueName: \"kubernetes.io/projected/606b6ef9-c443-4760-bf82-462f49e08d23-kube-api-access-86v5g\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.528863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.528837 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:00.545401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.537949 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:00.545660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.545639 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:00.892170 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:00.892128 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987e1d4c_33df_4a9d_8f31_6a292c2c8fff.slice/crio-557883352b7bbdafa9815e757cc23e3e13134fc2f961dd734d61583c1c5e156d WatchSource:0}: Error finding container 557883352b7bbdafa9815e757cc23e3e13134fc2f961dd734d61583c1c5e156d: Status 404 returned error can't find the container with id 557883352b7bbdafa9815e757cc23e3e13134fc2f961dd734d61583c1c5e156d Nov 02 20:15:00.893201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:00.893169 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692"] Nov 02 20:15:01.019364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.019334 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9"] Nov 02 20:15:01.020948 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:01.020915 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd920b9b6_8c49_4320_9ca8_fbeb37c75703.slice/crio-53b473a2b7bacf4fa04da0c8eae9ce2aa1cc42a4f2c06b9c89d5e517f8a2328c WatchSource:0}: Error finding container 53b473a2b7bacf4fa04da0c8eae9ce2aa1cc42a4f2c06b9c89d5e517f8a2328c: Status 404 returned error can't find the container with id 53b473a2b7bacf4fa04da0c8eae9ce2aa1cc42a4f2c06b9c89d5e517f8a2328c Nov 02 20:15:01.043939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.043909 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q"] Nov 02 20:15:01.044087 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:01.044059 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba91842_792f_45bc_831a_60dd4419cdcd.slice/crio-4daa3dd62d259c4aeb3ac21ecbc4d4d29630ffa3ba59d255908753823821300e WatchSource:0}: Error finding container 4daa3dd62d259c4aeb3ac21ecbc4d4d29630ffa3ba59d255908753823821300e: Status 404 returned error can't find the container with id 4daa3dd62d259c4aeb3ac21ecbc4d4d29630ffa3ba59d255908753823821300e Nov 02 20:15:01.052093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.052064 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4"] Nov 02 20:15:01.057267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.057244 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90fb2618-s29b4"] Nov 02 20:15:01.095751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.095721 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49"] Nov 02 20:15:01.096138 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:01.096114 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59041d79_d3f5_42d6_bf89_633f228cd747.slice/crio-38d30e34f718f508fe927e654387e8c414431e47938ce52dceb76e454199d343 WatchSource:0}: Error finding container 38d30e34f718f508fe927e654387e8c414431e47938ce52dceb76e454199d343: Status 404 returned error can't find the container with id 38d30e34f718f508fe927e654387e8c414431e47938ce52dceb76e454199d343 Nov 02 20:15:01.120528 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:01.120500 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189e8ce6_3ae1_43e0_a637_0300c3f9deff.slice/crio-9436521f97603e2ef76f57f71a9e61c32dd8ed730eb7b3792b049a9998b3f9ca WatchSource:0}: Error finding container 9436521f97603e2ef76f57f71a9e61c32dd8ed730eb7b3792b049a9998b3f9ca: Status 404 returned error can't find the container with id 9436521f97603e2ef76f57f71a9e61c32dd8ed730eb7b3792b049a9998b3f9ca Nov 02 20:15:01.121125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.121065 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl"] Nov 02 20:15:01.177660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.177627 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn"] Nov 02 20:15:01.229482 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:01.229441 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606b6ef9_c443_4760_bf82_462f49e08d23.slice/crio-d6a597003ea47614412aea82987d7384b2e6c30ff4b11b20c2582f04bf7f7387 WatchSource:0}: Error finding container d6a597003ea47614412aea82987d7384b2e6c30ff4b11b20c2582f04bf7f7387: Status 404 returned error can't find the container with id d6a597003ea47614412aea82987d7384b2e6c30ff4b11b20c2582f04bf7f7387 Nov 02 20:15:01.548578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.548527 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" event={"ID":"1ba91842-792f-45bc-831a-60dd4419cdcd","Type":"ContainerStarted","Data":"c063e61596acbf2972b41bcf362c85e6cb5aef5ab1e9203474e61ed3c335ff6a"} Nov 02 20:15:01.548725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.548595 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" event={"ID":"1ba91842-792f-45bc-831a-60dd4419cdcd","Type":"ContainerStarted","Data":"4daa3dd62d259c4aeb3ac21ecbc4d4d29630ffa3ba59d255908753823821300e"} Nov 02 20:15:01.551274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.551217 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" event={"ID":"606b6ef9-c443-4760-bf82-462f49e08d23","Type":"ContainerStarted","Data":"d6a597003ea47614412aea82987d7384b2e6c30ff4b11b20c2582f04bf7f7387"} Nov 02 20:15:01.553063 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.553002 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" event={"ID":"59041d79-d3f5-42d6-bf89-633f228cd747","Type":"ContainerStarted","Data":"38d30e34f718f508fe927e654387e8c414431e47938ce52dceb76e454199d343"} Nov 02 20:15:01.554932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.554904 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" event={"ID":"189e8ce6-3ae1-43e0-a637-0300c3f9deff","Type":"ContainerStarted","Data":"9436521f97603e2ef76f57f71a9e61c32dd8ed730eb7b3792b049a9998b3f9ca"} Nov 02 20:15:01.556605 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.556581 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" event={"ID":"d920b9b6-8c49-4320-9ca8-fbeb37c75703","Type":"ContainerStarted","Data":"53b473a2b7bacf4fa04da0c8eae9ce2aa1cc42a4f2c06b9c89d5e517f8a2328c"} Nov 02 20:15:01.558033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.558007 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" event={"ID":"987e1d4c-33df-4a9d-8f31-6a292c2c8fff","Type":"ContainerStarted","Data":"557883352b7bbdafa9815e757cc23e3e13134fc2f961dd734d61583c1c5e156d"} Nov 02 20:15:01.625865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.625790 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cc0532-f285-4081-9e42-11c206442d17" path="/var/lib/kubelet/pods/46cc0532-f285-4081-9e42-11c206442d17/volumes" Nov 02 20:15:01.657152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:01.656914 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" podStartSLOduration=1.656897217 podStartE2EDuration="1.656897217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:01.653647853 +0000 UTC m=+1600.170420439" watchObservedRunningTime="2026-11-02 20:15:01.656897217 +0000 UTC m=+1600.173669809" Nov 02 20:15:02.569442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.569405 5800 generic.go:356] "Generic (PLEG): container finished" podID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerID="cc5dfeb9758369d1cba1319c642c3ad2c3260ebe91b715ea0bf2213d98ae8df1" exitCode=60 Nov 02 20:15:02.569888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.569502 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" event={"ID":"d920b9b6-8c49-4320-9ca8-fbeb37c75703","Type":"ContainerDied","Data":"cc5dfeb9758369d1cba1319c642c3ad2c3260ebe91b715ea0bf2213d98ae8df1"} Nov 02 20:15:02.569888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.569855 5800 scope.go:122] "RemoveContainer" containerID="cc5dfeb9758369d1cba1319c642c3ad2c3260ebe91b715ea0bf2213d98ae8df1" Nov 02 20:15:02.572660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.572638 5800 generic.go:356] "Generic (PLEG): container finished" podID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerID="eefc64ababd9f46945baa26a06610cb4205d818b639bb138a350bae5d4045e2d" exitCode=60 Nov 02 20:15:02.572760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.572729 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" event={"ID":"987e1d4c-33df-4a9d-8f31-6a292c2c8fff","Type":"ContainerDied","Data":"eefc64ababd9f46945baa26a06610cb4205d818b639bb138a350bae5d4045e2d"} Nov 02 20:15:02.573132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.573094 5800 scope.go:122] "RemoveContainer" containerID="eefc64ababd9f46945baa26a06610cb4205d818b639bb138a350bae5d4045e2d" Nov 02 20:15:02.578373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.578359 5800 generic.go:356] "Generic (PLEG): container finished" podID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerID="c063e61596acbf2972b41bcf362c85e6cb5aef5ab1e9203474e61ed3c335ff6a" exitCode=60 Nov 02 20:15:02.578466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.578442 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" event={"ID":"1ba91842-792f-45bc-831a-60dd4419cdcd","Type":"ContainerDied","Data":"c063e61596acbf2972b41bcf362c85e6cb5aef5ab1e9203474e61ed3c335ff6a"} Nov 02 20:15:02.578671 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.578658 5800 scope.go:122] "RemoveContainer" containerID="c063e61596acbf2972b41bcf362c85e6cb5aef5ab1e9203474e61ed3c335ff6a" Nov 02 20:15:02.580181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.580154 5800 generic.go:356] "Generic (PLEG): container finished" podID="606b6ef9-c443-4760-bf82-462f49e08d23" containerID="632c9a5ff7829151be4c919d2e08050410f007868f3718e4249182323da6c16d" exitCode=60 Nov 02 20:15:02.580300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.580232 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" event={"ID":"606b6ef9-c443-4760-bf82-462f49e08d23","Type":"ContainerDied","Data":"632c9a5ff7829151be4c919d2e08050410f007868f3718e4249182323da6c16d"} Nov 02 20:15:02.580517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.580498 5800 scope.go:122] "RemoveContainer" containerID="632c9a5ff7829151be4c919d2e08050410f007868f3718e4249182323da6c16d" Nov 02 20:15:02.582496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.581806 5800 generic.go:356] "Generic (PLEG): container finished" podID="59041d79-d3f5-42d6-bf89-633f228cd747" containerID="b4d430fd9b9278233cac1eb08571c16fb3cdbd75f45e6bcd4026d7750194886e" exitCode=60 Nov 02 20:15:02.582496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.581872 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" event={"ID":"59041d79-d3f5-42d6-bf89-633f228cd747","Type":"ContainerDied","Data":"b4d430fd9b9278233cac1eb08571c16fb3cdbd75f45e6bcd4026d7750194886e"} Nov 02 20:15:02.582496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.582105 5800 scope.go:122] "RemoveContainer" containerID="b4d430fd9b9278233cac1eb08571c16fb3cdbd75f45e6bcd4026d7750194886e" Nov 02 20:15:02.584449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.584425 5800 generic.go:356] "Generic (PLEG): container finished" podID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerID="72c1dd3daf6ace7908b95f302dabdcf4e0fae327de8a6760df0e4be2fc62ce2d" exitCode=60 Nov 02 20:15:02.584530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.584490 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" event={"ID":"189e8ce6-3ae1-43e0-a637-0300c3f9deff","Type":"ContainerDied","Data":"72c1dd3daf6ace7908b95f302dabdcf4e0fae327de8a6760df0e4be2fc62ce2d"} Nov 02 20:15:02.584820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:02.584769 5800 scope.go:122] "RemoveContainer" containerID="72c1dd3daf6ace7908b95f302dabdcf4e0fae327de8a6760df0e4be2fc62ce2d" Nov 02 20:15:03.596614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.596569 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" event={"ID":"606b6ef9-c443-4760-bf82-462f49e08d23","Type":"ContainerStarted","Data":"d7a56dd636a90bc26316edc0227853109d019e86ab7c902d4d6c9cb2700cefa7"} Nov 02 20:15:03.598618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.598586 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" event={"ID":"59041d79-d3f5-42d6-bf89-633f228cd747","Type":"ContainerStarted","Data":"ab0234a27d99ec49e320001ba202384664342ffef3f531e4b28744278849076f"} Nov 02 20:15:03.600536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.600508 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" event={"ID":"189e8ce6-3ae1-43e0-a637-0300c3f9deff","Type":"ContainerStarted","Data":"0ae186d57b2a4d37aebae7cc34b6f230cd22187666bcc1b24ed3f3f863cdde7f"} Nov 02 20:15:03.602486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.602458 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" event={"ID":"d920b9b6-8c49-4320-9ca8-fbeb37c75703","Type":"ContainerStarted","Data":"bf24a51e46c653438de8c26eea6c0be7df820130aa9aaf24769b89dbaa11e3c9"} Nov 02 20:15:03.604361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.604335 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" event={"ID":"987e1d4c-33df-4a9d-8f31-6a292c2c8fff","Type":"ContainerStarted","Data":"8f1926116ff2fe10e9a25aaca3760530fd6134d58bc595931fd40d0fcedf59d0"} Nov 02 20:15:03.606184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.606161 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" event={"ID":"1ba91842-792f-45bc-831a-60dd4419cdcd","Type":"ContainerStarted","Data":"098ddfc72a459cfb9cafb97611bc76b26e64526f37c7cde4448c9f90bff99dd6"} Nov 02 20:15:03.706950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.706893 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" podStartSLOduration=3.7068796969999998 podStartE2EDuration="3.706879697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.702257275 +0000 UTC m=+1602.219029858" watchObservedRunningTime="2026-11-02 20:15:03.706879697 +0000 UTC m=+1602.223652281" Nov 02 20:15:03.918506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:03.918454 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" podStartSLOduration=3.918439965 podStartE2EDuration="3.918439965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:03.91535587 +0000 UTC m=+1602.432128458" watchObservedRunningTime="2026-11-02 20:15:03.918439965 +0000 UTC m=+1602.435212550" Nov 02 20:15:04.022143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.022091 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" podStartSLOduration=4.022077 podStartE2EDuration="4.022077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:04.02123436 +0000 UTC m=+1602.538006946" watchObservedRunningTime="2026-11-02 20:15:04.022077 +0000 UTC m=+1602.538849583" Nov 02 20:15:04.126877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.126649 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" podStartSLOduration=4.126633098 podStartE2EDuration="4.126633098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:04.126142251 +0000 UTC m=+1602.642914837" watchObservedRunningTime="2026-11-02 20:15:04.126633098 +0000 UTC m=+1602.643405684" Nov 02 20:15:04.232842 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.232777 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" podStartSLOduration=4.232761348 podStartE2EDuration="4.232761348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:04.230836473 +0000 UTC m=+1602.747609059" watchObservedRunningTime="2026-11-02 20:15:04.232761348 +0000 UTC m=+1602.749533933" Nov 02 20:15:04.618095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.618059 5800 generic.go:356] "Generic (PLEG): container finished" podID="59041d79-d3f5-42d6-bf89-633f228cd747" containerID="ab0234a27d99ec49e320001ba202384664342ffef3f531e4b28744278849076f" exitCode=60 Nov 02 20:15:04.618617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.618145 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" event={"ID":"59041d79-d3f5-42d6-bf89-633f228cd747","Type":"ContainerDied","Data":"ab0234a27d99ec49e320001ba202384664342ffef3f531e4b28744278849076f"} Nov 02 20:15:04.618617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.618213 5800 scope.go:122] "RemoveContainer" containerID="b4d430fd9b9278233cac1eb08571c16fb3cdbd75f45e6bcd4026d7750194886e" Nov 02 20:15:04.618617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.618498 5800 scope.go:122] "RemoveContainer" containerID="ab0234a27d99ec49e320001ba202384664342ffef3f531e4b28744278849076f" Nov 02 20:15:04.618932 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:04.618908 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49_odh-ai-gateway-infra(59041d79-d3f5-42d6-bf89-633f228cd747)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" Nov 02 20:15:04.620402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.620382 5800 generic.go:356] "Generic (PLEG): container finished" podID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerID="0ae186d57b2a4d37aebae7cc34b6f230cd22187666bcc1b24ed3f3f863cdde7f" exitCode=60 Nov 02 20:15:04.620501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.620407 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" event={"ID":"189e8ce6-3ae1-43e0-a637-0300c3f9deff","Type":"ContainerDied","Data":"0ae186d57b2a4d37aebae7cc34b6f230cd22187666bcc1b24ed3f3f863cdde7f"} Nov 02 20:15:04.620769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.620752 5800 scope.go:122] "RemoveContainer" containerID="0ae186d57b2a4d37aebae7cc34b6f230cd22187666bcc1b24ed3f3f863cdde7f" Nov 02 20:15:04.621123 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:04.621099 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl_odh-ai-gateway-infra(189e8ce6-3ae1-43e0-a637-0300c3f9deff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" Nov 02 20:15:04.622611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.622589 5800 generic.go:356] "Generic (PLEG): container finished" podID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerID="bf24a51e46c653438de8c26eea6c0be7df820130aa9aaf24769b89dbaa11e3c9" exitCode=60 Nov 02 20:15:04.622713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.622648 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" event={"ID":"d920b9b6-8c49-4320-9ca8-fbeb37c75703","Type":"ContainerDied","Data":"bf24a51e46c653438de8c26eea6c0be7df820130aa9aaf24769b89dbaa11e3c9"} Nov 02 20:15:04.622956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.622938 5800 scope.go:122] "RemoveContainer" containerID="bf24a51e46c653438de8c26eea6c0be7df820130aa9aaf24769b89dbaa11e3c9" Nov 02 20:15:04.623231 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:04.623210 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9_odh-ai-gateway-infra(d920b9b6-8c49-4320-9ca8-fbeb37c75703)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" Nov 02 20:15:04.624965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.624942 5800 generic.go:356] "Generic (PLEG): container finished" podID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerID="8f1926116ff2fe10e9a25aaca3760530fd6134d58bc595931fd40d0fcedf59d0" exitCode=60 Nov 02 20:15:04.625037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.625000 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" event={"ID":"987e1d4c-33df-4a9d-8f31-6a292c2c8fff","Type":"ContainerDied","Data":"8f1926116ff2fe10e9a25aaca3760530fd6134d58bc595931fd40d0fcedf59d0"} Nov 02 20:15:04.625277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.625259 5800 scope.go:122] "RemoveContainer" containerID="8f1926116ff2fe10e9a25aaca3760530fd6134d58bc595931fd40d0fcedf59d0" Nov 02 20:15:04.625677 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:04.625653 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692_odh-ai-gateway-infra(987e1d4c-33df-4a9d-8f31-6a292c2c8fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" Nov 02 20:15:04.627372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.627351 5800 generic.go:356] "Generic (PLEG): container finished" podID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerID="098ddfc72a459cfb9cafb97611bc76b26e64526f37c7cde4448c9f90bff99dd6" exitCode=60 Nov 02 20:15:04.627460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.627424 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" event={"ID":"1ba91842-792f-45bc-831a-60dd4419cdcd","Type":"ContainerDied","Data":"098ddfc72a459cfb9cafb97611bc76b26e64526f37c7cde4448c9f90bff99dd6"} Nov 02 20:15:04.627691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.627677 5800 scope.go:122] "RemoveContainer" containerID="098ddfc72a459cfb9cafb97611bc76b26e64526f37c7cde4448c9f90bff99dd6" Nov 02 20:15:04.627893 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:04.627877 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q_odh-ai-gateway-infra(1ba91842-792f-45bc-831a-60dd4419cdcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" Nov 02 20:15:04.629687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.629669 5800 generic.go:356] "Generic (PLEG): container finished" podID="606b6ef9-c443-4760-bf82-462f49e08d23" containerID="d7a56dd636a90bc26316edc0227853109d019e86ab7c902d4d6c9cb2700cefa7" exitCode=60 Nov 02 20:15:04.629765 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.629698 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" event={"ID":"606b6ef9-c443-4760-bf82-462f49e08d23","Type":"ContainerDied","Data":"d7a56dd636a90bc26316edc0227853109d019e86ab7c902d4d6c9cb2700cefa7"} Nov 02 20:15:04.629943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.629931 5800 scope.go:122] "RemoveContainer" containerID="d7a56dd636a90bc26316edc0227853109d019e86ab7c902d4d6c9cb2700cefa7" Nov 02 20:15:04.630148 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:04.630131 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn_odh-ai-gateway-infra(606b6ef9-c443-4760-bf82-462f49e08d23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" Nov 02 20:15:04.639387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.639362 5800 scope.go:122] "RemoveContainer" containerID="72c1dd3daf6ace7908b95f302dabdcf4e0fae327de8a6760df0e4be2fc62ce2d" Nov 02 20:15:04.656271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.656239 5800 scope.go:122] "RemoveContainer" containerID="cc5dfeb9758369d1cba1319c642c3ad2c3260ebe91b715ea0bf2213d98ae8df1" Nov 02 20:15:04.675687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.675662 5800 scope.go:122] "RemoveContainer" containerID="eefc64ababd9f46945baa26a06610cb4205d818b639bb138a350bae5d4045e2d" Nov 02 20:15:04.690474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.690450 5800 scope.go:122] "RemoveContainer" containerID="c063e61596acbf2972b41bcf362c85e6cb5aef5ab1e9203474e61ed3c335ff6a" Nov 02 20:15:04.705092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:04.705069 5800 scope.go:122] "RemoveContainer" containerID="632c9a5ff7829151be4c919d2e08050410f007868f3718e4249182323da6c16d" Nov 02 20:15:05.685729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:05.685699 5800 scope.go:122] "RemoveContainer" containerID="8f1926116ff2fe10e9a25aaca3760530fd6134d58bc595931fd40d0fcedf59d0" Nov 02 20:15:05.686659 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:05.686628 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692_odh-ai-gateway-infra(987e1d4c-33df-4a9d-8f31-6a292c2c8fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" Nov 02 20:15:05.701509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:05.699592 5800 scope.go:122] "RemoveContainer" containerID="098ddfc72a459cfb9cafb97611bc76b26e64526f37c7cde4448c9f90bff99dd6" Nov 02 20:15:05.701509 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:05.699979 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q_odh-ai-gateway-infra(1ba91842-792f-45bc-831a-60dd4419cdcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" Nov 02 20:15:05.708303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:05.708255 5800 scope.go:122] "RemoveContainer" containerID="d7a56dd636a90bc26316edc0227853109d019e86ab7c902d4d6c9cb2700cefa7" Nov 02 20:15:05.708758 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:05.708722 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn_odh-ai-gateway-infra(606b6ef9-c443-4760-bf82-462f49e08d23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" Nov 02 20:15:05.712528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:05.712502 5800 scope.go:122] "RemoveContainer" containerID="ab0234a27d99ec49e320001ba202384664342ffef3f531e4b28744278849076f" Nov 02 20:15:05.712893 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:05.712870 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49_odh-ai-gateway-infra(59041d79-d3f5-42d6-bf89-633f228cd747)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" Nov 02 20:15:05.715269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:05.715247 5800 scope.go:122] "RemoveContainer" containerID="0ae186d57b2a4d37aebae7cc34b6f230cd22187666bcc1b24ed3f3f863cdde7f" Nov 02 20:15:05.715631 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:05.715610 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl_odh-ai-gateway-infra(189e8ce6-3ae1-43e0-a637-0300c3f9deff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" Nov 02 20:15:05.720254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:05.720236 5800 scope.go:122] "RemoveContainer" containerID="bf24a51e46c653438de8c26eea6c0be7df820130aa9aaf24769b89dbaa11e3c9" Nov 02 20:15:05.720614 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:05.720590 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9_odh-ai-gateway-infra(d920b9b6-8c49-4320-9ca8-fbeb37c75703)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" Nov 02 20:15:07.308395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.308360 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q"] Nov 02 20:15:07.610376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.606299 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:07.695966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.695933 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba91842-792f-45bc-831a-60dd4419cdcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba91842-792f-45bc-831a-60dd4419cdcd-maas-api-service-ca\") pod \"1ba91842-792f-45bc-831a-60dd4419cdcd\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " Nov 02 20:15:07.696155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.696040 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba91842-792f-45bc-831a-60dd4419cdcd-kube-api-access-98tf7\" (UniqueName: \"kubernetes.io/projected/1ba91842-792f-45bc-831a-60dd4419cdcd-kube-api-access-98tf7\") pod \"1ba91842-792f-45bc-831a-60dd4419cdcd\" (UID: \"1ba91842-792f-45bc-831a-60dd4419cdcd\") " Nov 02 20:15:07.696334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.696307 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba91842-792f-45bc-831a-60dd4419cdcd-maas-api-service-ca" pod "1ba91842-792f-45bc-831a-60dd4419cdcd" (UID: "1ba91842-792f-45bc-831a-60dd4419cdcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:07.696619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.696606 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba91842-792f-45bc-831a-60dd4419cdcd-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:07.698646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.698613 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba91842-792f-45bc-831a-60dd4419cdcd-kube-api-access-98tf7" pod "1ba91842-792f-45bc-831a-60dd4419cdcd" (UID: "1ba91842-792f-45bc-831a-60dd4419cdcd"). InnerVolumeSpecName "kube-api-access-98tf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:07.754117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.744619 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" event={"ID":"1ba91842-792f-45bc-831a-60dd4419cdcd","Type":"ContainerDied","Data":"4daa3dd62d259c4aeb3ac21ecbc4d4d29630ffa3ba59d255908753823821300e"} Nov 02 20:15:07.754117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.744693 5800 scope.go:122] "RemoveContainer" containerID="098ddfc72a459cfb9cafb97611bc76b26e64526f37c7cde4448c9f90bff99dd6" Nov 02 20:15:07.754117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.745434 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q" Nov 02 20:15:07.798136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.798107 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98tf7\" (UniqueName: \"kubernetes.io/projected/1ba91842-792f-45bc-831a-60dd4419cdcd-kube-api-access-98tf7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:07.906193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.906103 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q"] Nov 02 20:15:07.910232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:07.910202 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-45c182-29894175-lrh5q"] Nov 02 20:15:09.624125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:09.624093 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" path="/var/lib/kubelet/pods/1ba91842-792f-45bc-831a-60dd4419cdcd/volumes" Nov 02 20:15:12.172004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:12.171963 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps"] Nov 02 20:15:12.172423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:12.172162 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" containerID="cri-o://e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7" gracePeriod=30 Nov 02 20:15:12.219364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:12.219327 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c"] Nov 02 20:15:12.219554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:12.219532 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" containerID="cri-o://47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e" gracePeriod=30 Nov 02 20:15:14.614865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:14.614834 5800 scope.go:122] "RemoveContainer" containerID="d7a56dd636a90bc26316edc0227853109d019e86ab7c902d4d6c9cb2700cefa7" Nov 02 20:15:15.615339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:15.615307 5800 scope.go:122] "RemoveContainer" containerID="bf24a51e46c653438de8c26eea6c0be7df820130aa9aaf24769b89dbaa11e3c9" Nov 02 20:15:15.615755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:15.615475 5800 scope.go:122] "RemoveContainer" containerID="8f1926116ff2fe10e9a25aaca3760530fd6134d58bc595931fd40d0fcedf59d0" Nov 02 20:15:15.616023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:15.616005 5800 scope.go:122] "RemoveContainer" containerID="ab0234a27d99ec49e320001ba202384664342ffef3f531e4b28744278849076f" Nov 02 20:15:15.839161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:15.839079 5800 generic.go:356] "Generic (PLEG): container finished" podID="606b6ef9-c443-4760-bf82-462f49e08d23" containerID="0def508db1a7f68d638dd789be7ef54c5e951f6b684f19ec56d23d7fd9460076" exitCode=60 Nov 02 20:15:15.839161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:15.839106 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" event={"ID":"606b6ef9-c443-4760-bf82-462f49e08d23","Type":"ContainerDied","Data":"0def508db1a7f68d638dd789be7ef54c5e951f6b684f19ec56d23d7fd9460076"} Nov 02 20:15:15.840016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:15.839178 5800 scope.go:122] "RemoveContainer" containerID="d7a56dd636a90bc26316edc0227853109d019e86ab7c902d4d6c9cb2700cefa7" Nov 02 20:15:15.840016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:15.839632 5800 scope.go:122] "RemoveContainer" containerID="0def508db1a7f68d638dd789be7ef54c5e951f6b684f19ec56d23d7fd9460076" Nov 02 20:15:15.840116 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:15.840020 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn_odh-ai-gateway-infra(606b6ef9-c443-4760-bf82-462f49e08d23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" Nov 02 20:15:16.615452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.615415 5800 scope.go:122] "RemoveContainer" containerID="0ae186d57b2a4d37aebae7cc34b6f230cd22187666bcc1b24ed3f3f863cdde7f" Nov 02 20:15:16.853803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.853768 5800 generic.go:356] "Generic (PLEG): container finished" podID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerID="9389ffdfba9e1298b70d865d13150b26a82145e923a26a51a762f7e55c1e42e4" exitCode=60 Nov 02 20:15:16.853988 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.853810 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" event={"ID":"d920b9b6-8c49-4320-9ca8-fbeb37c75703","Type":"ContainerDied","Data":"9389ffdfba9e1298b70d865d13150b26a82145e923a26a51a762f7e55c1e42e4"} Nov 02 20:15:16.853988 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.853870 5800 scope.go:122] "RemoveContainer" containerID="bf24a51e46c653438de8c26eea6c0be7df820130aa9aaf24769b89dbaa11e3c9" Nov 02 20:15:16.854375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.854354 5800 scope.go:122] "RemoveContainer" containerID="9389ffdfba9e1298b70d865d13150b26a82145e923a26a51a762f7e55c1e42e4" Nov 02 20:15:16.854779 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:16.854748 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9_odh-ai-gateway-infra(d920b9b6-8c49-4320-9ca8-fbeb37c75703)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" Nov 02 20:15:16.856677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.856653 5800 generic.go:356] "Generic (PLEG): container finished" podID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerID="d6438d26ad6c66cd508741af6bafc3727b8bcb1c93c5abb9376a1708b4d0f62f" exitCode=60 Nov 02 20:15:16.856781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.856726 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" event={"ID":"987e1d4c-33df-4a9d-8f31-6a292c2c8fff","Type":"ContainerDied","Data":"d6438d26ad6c66cd508741af6bafc3727b8bcb1c93c5abb9376a1708b4d0f62f"} Nov 02 20:15:16.857232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.857213 5800 scope.go:122] "RemoveContainer" containerID="d6438d26ad6c66cd508741af6bafc3727b8bcb1c93c5abb9376a1708b4d0f62f" Nov 02 20:15:16.857843 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:16.857618 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692_odh-ai-gateway-infra(987e1d4c-33df-4a9d-8f31-6a292c2c8fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" Nov 02 20:15:16.861124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.861097 5800 generic.go:356] "Generic (PLEG): container finished" podID="59041d79-d3f5-42d6-bf89-633f228cd747" containerID="0932dc7daae733298d57fb5d7b0f5c7a58cfc2518c38b447a68279756efd4882" exitCode=60 Nov 02 20:15:16.861354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.861338 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" event={"ID":"59041d79-d3f5-42d6-bf89-633f228cd747","Type":"ContainerDied","Data":"0932dc7daae733298d57fb5d7b0f5c7a58cfc2518c38b447a68279756efd4882"} Nov 02 20:15:16.861803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.861785 5800 scope.go:122] "RemoveContainer" containerID="0932dc7daae733298d57fb5d7b0f5c7a58cfc2518c38b447a68279756efd4882" Nov 02 20:15:16.862194 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:16.862173 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49_odh-ai-gateway-infra(59041d79-d3f5-42d6-bf89-633f228cd747)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" Nov 02 20:15:16.878417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.878395 5800 scope.go:122] "RemoveContainer" containerID="8f1926116ff2fe10e9a25aaca3760530fd6134d58bc595931fd40d0fcedf59d0" Nov 02 20:15:16.892157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.892137 5800 scope.go:122] "RemoveContainer" containerID="ab0234a27d99ec49e320001ba202384664342ffef3f531e4b28744278849076f" Nov 02 20:15:16.965591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:16.965538 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn"] Nov 02 20:15:17.352558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.352530 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:17.487101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.486973 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606b6ef9-c443-4760-bf82-462f49e08d23-kube-api-access-86v5g\" (UniqueName: \"kubernetes.io/projected/606b6ef9-c443-4760-bf82-462f49e08d23-kube-api-access-86v5g\") pod \"606b6ef9-c443-4760-bf82-462f49e08d23\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " Nov 02 20:15:17.487878 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.487846 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606b6ef9-c443-4760-bf82-462f49e08d23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b6ef9-c443-4760-bf82-462f49e08d23-maas-api-service-ca\") pod \"606b6ef9-c443-4760-bf82-462f49e08d23\" (UID: \"606b6ef9-c443-4760-bf82-462f49e08d23\") " Nov 02 20:15:17.488315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.488275 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606b6ef9-c443-4760-bf82-462f49e08d23-maas-api-service-ca" pod "606b6ef9-c443-4760-bf82-462f49e08d23" (UID: "606b6ef9-c443-4760-bf82-462f49e08d23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:17.489129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.489105 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/606b6ef9-c443-4760-bf82-462f49e08d23-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.493571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.493490 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606b6ef9-c443-4760-bf82-462f49e08d23-kube-api-access-86v5g" pod "606b6ef9-c443-4760-bf82-462f49e08d23" (UID: "606b6ef9-c443-4760-bf82-462f49e08d23"). InnerVolumeSpecName "kube-api-access-86v5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:17.523660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.523633 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:15:17.592576 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.592545 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-certs\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.592729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.592663 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26e053c-6975-4224-ac62-9a054e216d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26e053c-6975-4224-ac62-9a054e216d84-istiod-ca-cert\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.592729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.592702 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-credential-socket\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.592871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.592850 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-certs" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:17.592929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.592880 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d26e053c-6975-4224-ac62-9a054e216d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26e053c-6975-4224-ac62-9a054e216d84-istio-podinfo\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.592983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.592966 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-istio-token\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-istio-token\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.593041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593006 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-envoy\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.593102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593046 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-kube-api-access-vlmzb\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-kube-api-access-vlmzb\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.593102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593069 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-credential-socket" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:17.593102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593081 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-data\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.593255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593147 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26e053c-6975-4224-ac62-9a054e216d84-istiod-ca-cert" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:17.593255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593156 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-socket\") pod \"d26e053c-6975-4224-ac62-9a054e216d84\" (UID: \"d26e053c-6975-4224-ac62-9a054e216d84\") " Nov 02 20:15:17.593441 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593270 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-data" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:17.593501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593366 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-socket" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:17.593976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593861 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.593976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593890 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.593976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593906 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.593976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593920 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26e053c-6975-4224-ac62-9a054e216d84-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.593976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593936 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.593976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.593951 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86v5g\" (UniqueName: \"kubernetes.io/projected/606b6ef9-c443-4760-bf82-462f49e08d23-kube-api-access-86v5g\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.595646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.595616 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-kube-api-access-vlmzb" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "kube-api-access-vlmzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:17.595923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.595892 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d26e053c-6975-4224-ac62-9a054e216d84-istio-podinfo" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:17.596124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.596101 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-istio-token" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:17.596349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.596327 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-envoy" pod "d26e053c-6975-4224-ac62-9a054e216d84" (UID: "d26e053c-6975-4224-ac62-9a054e216d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:17.695869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.695828 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26e053c-6975-4224-ac62-9a054e216d84-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.695869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.695868 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.696416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.695884 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26e053c-6975-4224-ac62-9a054e216d84-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.696416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.695899 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlmzb\" (UniqueName: \"kubernetes.io/projected/d26e053c-6975-4224-ac62-9a054e216d84-kube-api-access-vlmzb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:17.856912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.856888 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:15:17.875453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.875418 5800 generic.go:356] "Generic (PLEG): container finished" podID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerID="e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7" exitCode=0 Nov 02 20:15:17.875603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.875489 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" Nov 02 20:15:17.875603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.875518 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" event={"ID":"10e31a97-85a6-4372-9d49-9a59d5da22b4","Type":"ContainerDied","Data":"e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7"} Nov 02 20:15:17.875603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.875584 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps" event={"ID":"10e31a97-85a6-4372-9d49-9a59d5da22b4","Type":"ContainerDied","Data":"003f3688052486eccd0ed80fdd4a2206b35f90407c3b22209301c5ee4e4e8e5d"} Nov 02 20:15:17.875763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.875618 5800 scope.go:122] "RemoveContainer" containerID="e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7" Nov 02 20:15:17.877318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.877296 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" Nov 02 20:15:17.877318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.877308 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn" event={"ID":"606b6ef9-c443-4760-bf82-462f49e08d23","Type":"ContainerDied","Data":"d6a597003ea47614412aea82987d7384b2e6c30ff4b11b20c2582f04bf7f7387"} Nov 02 20:15:17.880711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.880680 5800 generic.go:356] "Generic (PLEG): container finished" podID="d26e053c-6975-4224-ac62-9a054e216d84" containerID="47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e" exitCode=0 Nov 02 20:15:17.880828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.880713 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" event={"ID":"d26e053c-6975-4224-ac62-9a054e216d84","Type":"ContainerDied","Data":"47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e"} Nov 02 20:15:17.880828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.880740 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" Nov 02 20:15:17.880828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.880762 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c" event={"ID":"d26e053c-6975-4224-ac62-9a054e216d84","Type":"ContainerDied","Data":"90b15fdfe00c28f3be887d3fc842dfae069fe5b98463d3eb088bb4b66021a4b8"} Nov 02 20:15:17.882732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.882707 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" event={"ID":"189e8ce6-3ae1-43e0-a637-0300c3f9deff","Type":"ContainerStarted","Data":"cd0cdaeb7e4e3eb60e287830f49cd921eeec20d37c11c61eee67d5d56d95fd30"} Nov 02 20:15:17.892214 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.892118 5800 scope.go:122] "RemoveContainer" containerID="e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7" Nov 02 20:15:17.892720 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:17.892591 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7\": container with ID starting with e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7 not found: ID does not exist" containerID="e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7" Nov 02 20:15:17.892720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.892628 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7"} err="failed to get container status \"e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7\": rpc error: code = NotFound desc = could not find container \"e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7\": container with ID starting with e5cfaa1a0dd8d89d29215eb300a374df28c1c4441372592476bcce1503965ca7 not found: ID does not exist" Nov 02 20:15:17.892720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.892653 5800 scope.go:122] "RemoveContainer" containerID="0def508db1a7f68d638dd789be7ef54c5e951f6b684f19ec56d23d7fd9460076" Nov 02 20:15:17.903906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.903881 5800 scope.go:122] "RemoveContainer" containerID="47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e" Nov 02 20:15:17.914803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.914785 5800 scope.go:122] "RemoveContainer" containerID="47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e" Nov 02 20:15:17.915086 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:17.915069 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e\": container with ID starting with 47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e not found: ID does not exist" containerID="47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e" Nov 02 20:15:17.915140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:17.915095 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e"} err="failed to get container status \"47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e\": rpc error: code = NotFound desc = could not find container \"47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e\": container with ID starting with 47ecfb607a338403d870d340d313c3b585b07f18a3a4a392b475f5d0acdd960e not found: ID does not exist" Nov 02 20:15:18.000193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000153 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-data\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000231 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-token\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-token\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000327 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-credential-socket\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000377 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-certs\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000449 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-kube-api-access-lsnb7\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-kube-api-access-lsnb7\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000503 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e31a97-85a6-4372-9d49-9a59d5da22b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e31a97-85a6-4372-9d49-9a59d5da22b4-istiod-ca-cert\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000522 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-data" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:18.000767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000598 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-podinfo\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000698 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-socket\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.000897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.000876 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-envoy\") pod \"10e31a97-85a6-4372-9d49-9a59d5da22b4\" (UID: \"10e31a97-85a6-4372-9d49-9a59d5da22b4\") " Nov 02 20:15:18.001050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.001025 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-certs" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:18.002005 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.001977 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-socket" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:18.002532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.002504 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-credential-socket" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:18.003433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.003326 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.003433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.003351 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.003433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.003366 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.003433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.003382 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.003433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.003393 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-token" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.003433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.003422 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e31a97-85a6-4372-9d49-9a59d5da22b4-istiod-ca-cert" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.004073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.004047 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-envoy" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:18.006085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.006062 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-kube-api-access-lsnb7" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "kube-api-access-lsnb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.006753 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.006728 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-podinfo" pod "10e31a97-85a6-4372-9d49-9a59d5da22b4" (UID: "10e31a97-85a6-4372-9d49-9a59d5da22b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:18.015474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.015453 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9"] Nov 02 20:15:18.105056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.105022 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.105056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.105058 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.105308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.105075 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsnb7\" (UniqueName: \"kubernetes.io/projected/10e31a97-85a6-4372-9d49-9a59d5da22b4-kube-api-access-lsnb7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.105308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.105090 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e31a97-85a6-4372-9d49-9a59d5da22b4-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.105308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.105106 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e31a97-85a6-4372-9d49-9a59d5da22b4-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.107881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.107849 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692"] Nov 02 20:15:18.184377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.184348 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn"] Nov 02 20:15:18.205750 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.198564 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894175-gmntn"] Nov 02 20:15:18.222755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.222733 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:18.238803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.238429 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49"] Nov 02 20:15:18.308655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.308554 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d920b9b6-8c49-4320-9ca8-fbeb37c75703-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d920b9b6-8c49-4320-9ca8-fbeb37c75703-maas-api-service-ca\") pod \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " Nov 02 20:15:18.308832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.308692 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d920b9b6-8c49-4320-9ca8-fbeb37c75703-kube-api-access-q69h4\" (UniqueName: \"kubernetes.io/projected/d920b9b6-8c49-4320-9ca8-fbeb37c75703-kube-api-access-q69h4\") pod \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\" (UID: \"d920b9b6-8c49-4320-9ca8-fbeb37c75703\") " Nov 02 20:15:18.309736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.309704 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d920b9b6-8c49-4320-9ca8-fbeb37c75703-maas-api-service-ca" pod "d920b9b6-8c49-4320-9ca8-fbeb37c75703" (UID: "d920b9b6-8c49-4320-9ca8-fbeb37c75703"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.311941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.311758 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d920b9b6-8c49-4320-9ca8-fbeb37c75703-kube-api-access-q69h4" pod "d920b9b6-8c49-4320-9ca8-fbeb37c75703" (UID: "d920b9b6-8c49-4320-9ca8-fbeb37c75703"). InnerVolumeSpecName "kube-api-access-q69h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.342436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.342404 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c"] Nov 02 20:15:18.348763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.348734 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-n8r4c"] Nov 02 20:15:18.355179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.355147 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:18.410888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.410857 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d920b9b6-8c49-4320-9ca8-fbeb37c75703-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.410888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.410885 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69h4\" (UniqueName: \"kubernetes.io/projected/d920b9b6-8c49-4320-9ca8-fbeb37c75703-kube-api-access-q69h4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.446686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.446659 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:18.448517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.448487 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps"] Nov 02 20:15:18.457927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.457900 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-ddzps"] Nov 02 20:15:18.511622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.511576 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59041d79-d3f5-42d6-bf89-633f228cd747-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59041d79-d3f5-42d6-bf89-633f228cd747-maas-api-service-ca\") pod \"59041d79-d3f5-42d6-bf89-633f228cd747\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " Nov 02 20:15:18.511840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.511822 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59041d79-d3f5-42d6-bf89-633f228cd747-kube-api-access-nhwbb\" (UniqueName: \"kubernetes.io/projected/59041d79-d3f5-42d6-bf89-633f228cd747-kube-api-access-nhwbb\") pod \"59041d79-d3f5-42d6-bf89-633f228cd747\" (UID: \"59041d79-d3f5-42d6-bf89-633f228cd747\") " Nov 02 20:15:18.511911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.511896 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-kube-api-access-46st6\" (UniqueName: \"kubernetes.io/projected/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-kube-api-access-46st6\") pod \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " Nov 02 20:15:18.511972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.511950 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59041d79-d3f5-42d6-bf89-633f228cd747-maas-api-service-ca" pod "59041d79-d3f5-42d6-bf89-633f228cd747" (UID: "59041d79-d3f5-42d6-bf89-633f228cd747"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.512033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.511989 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-maas-api-service-ca\") pod \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\" (UID: \"987e1d4c-33df-4a9d-8f31-6a292c2c8fff\") " Nov 02 20:15:18.512340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.512267 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-maas-api-service-ca" pod "987e1d4c-33df-4a9d-8f31-6a292c2c8fff" (UID: "987e1d4c-33df-4a9d-8f31-6a292c2c8fff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:18.512669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.512649 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.512725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.512677 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59041d79-d3f5-42d6-bf89-633f228cd747-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.514115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.514095 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-kube-api-access-46st6" pod "987e1d4c-33df-4a9d-8f31-6a292c2c8fff" (UID: "987e1d4c-33df-4a9d-8f31-6a292c2c8fff"). InnerVolumeSpecName "kube-api-access-46st6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.514197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.514125 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59041d79-d3f5-42d6-bf89-633f228cd747-kube-api-access-nhwbb" pod "59041d79-d3f5-42d6-bf89-633f228cd747" (UID: "59041d79-d3f5-42d6-bf89-633f228cd747"). InnerVolumeSpecName "kube-api-access-nhwbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:18.613821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.613736 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhwbb\" (UniqueName: \"kubernetes.io/projected/59041d79-d3f5-42d6-bf89-633f228cd747-kube-api-access-nhwbb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.613821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.613766 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46st6\" (UniqueName: \"kubernetes.io/projected/987e1d4c-33df-4a9d-8f31-6a292c2c8fff-kube-api-access-46st6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:18.897391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.896916 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" event={"ID":"d920b9b6-8c49-4320-9ca8-fbeb37c75703","Type":"ContainerDied","Data":"53b473a2b7bacf4fa04da0c8eae9ce2aa1cc42a4f2c06b9c89d5e517f8a2328c"} Nov 02 20:15:18.897391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.896997 5800 scope.go:122] "RemoveContainer" containerID="9389ffdfba9e1298b70d865d13150b26a82145e923a26a51a762f7e55c1e42e4" Nov 02 20:15:18.898003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.897981 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9" Nov 02 20:15:18.900210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.900160 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" event={"ID":"987e1d4c-33df-4a9d-8f31-6a292c2c8fff","Type":"ContainerDied","Data":"557883352b7bbdafa9815e757cc23e3e13134fc2f961dd734d61583c1c5e156d"} Nov 02 20:15:18.900370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.900255 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692" Nov 02 20:15:18.919410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.919385 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" Nov 02 20:15:18.919547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.919429 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49" event={"ID":"59041d79-d3f5-42d6-bf89-633f228cd747","Type":"ContainerDied","Data":"38d30e34f718f508fe927e654387e8c414431e47938ce52dceb76e454199d343"} Nov 02 20:15:18.919901 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.919883 5800 scope.go:122] "RemoveContainer" containerID="d6438d26ad6c66cd508741af6bafc3727b8bcb1c93c5abb9376a1708b4d0f62f" Nov 02 20:15:18.922838 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.922812 5800 generic.go:356] "Generic (PLEG): container finished" podID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerID="cd0cdaeb7e4e3eb60e287830f49cd921eeec20d37c11c61eee67d5d56d95fd30" exitCode=60 Nov 02 20:15:18.922951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.922860 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" event={"ID":"189e8ce6-3ae1-43e0-a637-0300c3f9deff","Type":"ContainerDied","Data":"cd0cdaeb7e4e3eb60e287830f49cd921eeec20d37c11c61eee67d5d56d95fd30"} Nov 02 20:15:18.923365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.923343 5800 scope.go:122] "RemoveContainer" containerID="cd0cdaeb7e4e3eb60e287830f49cd921eeec20d37c11c61eee67d5d56d95fd30" Nov 02 20:15:18.923712 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:18.923689 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl_odh-ai-gateway-infra(189e8ce6-3ae1-43e0-a637-0300c3f9deff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" Nov 02 20:15:18.934566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.934538 5800 scope.go:122] "RemoveContainer" containerID="0932dc7daae733298d57fb5d7b0f5c7a58cfc2518c38b447a68279756efd4882" Nov 02 20:15:18.953914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:18.953881 5800 scope.go:122] "RemoveContainer" containerID="0ae186d57b2a4d37aebae7cc34b6f230cd22187666bcc1b24ed3f3f863cdde7f" Nov 02 20:15:19.084998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.084953 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl"] Nov 02 20:15:19.098415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.098379 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692"] Nov 02 20:15:19.109249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.109219 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894175-76692"] Nov 02 20:15:19.207210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.207180 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49"] Nov 02 20:15:19.210945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.210917 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08eeaa-29894175-2gt49"] Nov 02 20:15:19.315337 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.315300 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9"] Nov 02 20:15:19.347171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.342964 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3021d-29894175-nh6c9"] Nov 02 20:15:19.627656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.627625 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" path="/var/lib/kubelet/pods/10e31a97-85a6-4372-9d49-9a59d5da22b4/volumes" Nov 02 20:15:19.628115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.628102 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" path="/var/lib/kubelet/pods/59041d79-d3f5-42d6-bf89-633f228cd747/volumes" Nov 02 20:15:19.628563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.628549 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" path="/var/lib/kubelet/pods/606b6ef9-c443-4760-bf82-462f49e08d23/volumes" Nov 02 20:15:19.636661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.632737 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" path="/var/lib/kubelet/pods/987e1d4c-33df-4a9d-8f31-6a292c2c8fff/volumes" Nov 02 20:15:19.636661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.633577 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26e053c-6975-4224-ac62-9a054e216d84" path="/var/lib/kubelet/pods/d26e053c-6975-4224-ac62-9a054e216d84/volumes" Nov 02 20:15:19.636661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:19.634171 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" path="/var/lib/kubelet/pods/d920b9b6-8c49-4320-9ca8-fbeb37c75703/volumes" Nov 02 20:15:20.228020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.227997 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:20.285973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.285945 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189e8ce6-3ae1-43e0-a637-0300c3f9deff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189e8ce6-3ae1-43e0-a637-0300c3f9deff-maas-api-service-ca\") pod \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " Nov 02 20:15:20.286180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.286055 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189e8ce6-3ae1-43e0-a637-0300c3f9deff-kube-api-access-697zq\" (UniqueName: \"kubernetes.io/projected/189e8ce6-3ae1-43e0-a637-0300c3f9deff-kube-api-access-697zq\") pod \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\" (UID: \"189e8ce6-3ae1-43e0-a637-0300c3f9deff\") " Nov 02 20:15:20.286418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.286391 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189e8ce6-3ae1-43e0-a637-0300c3f9deff-maas-api-service-ca" pod "189e8ce6-3ae1-43e0-a637-0300c3f9deff" (UID: "189e8ce6-3ae1-43e0-a637-0300c3f9deff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:20.286718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.286701 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/189e8ce6-3ae1-43e0-a637-0300c3f9deff-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:20.288241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.288212 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189e8ce6-3ae1-43e0-a637-0300c3f9deff-kube-api-access-697zq" pod "189e8ce6-3ae1-43e0-a637-0300c3f9deff" (UID: "189e8ce6-3ae1-43e0-a637-0300c3f9deff"). InnerVolumeSpecName "kube-api-access-697zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:20.388596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.388564 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-697zq\" (UniqueName: \"kubernetes.io/projected/189e8ce6-3ae1-43e0-a637-0300c3f9deff-kube-api-access-697zq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:20.945882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.945849 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" Nov 02 20:15:20.946080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.945875 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl" event={"ID":"189e8ce6-3ae1-43e0-a637-0300c3f9deff","Type":"ContainerDied","Data":"9436521f97603e2ef76f57f71a9e61c32dd8ed730eb7b3792b049a9998b3f9ca"} Nov 02 20:15:20.946080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:20.945942 5800 scope.go:122] "RemoveContainer" containerID="cd0cdaeb7e4e3eb60e287830f49cd921eeec20d37c11c61eee67d5d56d95fd30" Nov 02 20:15:21.103994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:21.103959 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl"] Nov 02 20:15:21.108087 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:21.108058 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894175-tkjdl"] Nov 02 20:15:21.624471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:21.624438 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" path="/var/lib/kubelet/pods/189e8ce6-3ae1-43e0-a637-0300c3f9deff/volumes" Nov 02 20:15:22.328740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.328705 5800 scope.go:122] "RemoveContainer" containerID="7b86b6842f385c1c46310633314c367e04913b6d56e229c488141ed282e39a44" Nov 02 20:15:22.339852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.339824 5800 scope.go:122] "RemoveContainer" containerID="4ec0cf4981afcd514219e690271419bc51591194c835538998a7b63469230649" Nov 02 20:15:22.351996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.351973 5800 scope.go:122] "RemoveContainer" containerID="db8b4f9ec1551cee712153b3edc08a760a1fab54d64e7319b2b8b7c609e443b8" Nov 02 20:15:22.364267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.364241 5800 scope.go:122] "RemoveContainer" containerID="e8776c9176940c3b43857a78c52d72a2f05e6d981e0f28b14e85175088491d2e" Nov 02 20:15:22.376812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.376784 5800 scope.go:122] "RemoveContainer" containerID="4ebc443e112470a7b20c470fb28a3a48ccca53311576013eb886e7fe333f60d2" Nov 02 20:15:22.386650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.386633 5800 scope.go:122] "RemoveContainer" containerID="d0ff408af4fcb89a21c1b3ad8e7690cb95da3df9b046d4d4257b711025435280" Nov 02 20:15:22.699263 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:22.699162 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:15:22.889733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.889702 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf"] Nov 02 20:15:22.992534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:22.992433 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq"] Nov 02 20:15:23.149119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:23.149073 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4"] Nov 02 20:15:23.149389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:23.149354 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" containerID="cri-o://3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1" gracePeriod=30 Nov 02 20:15:23.155032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:23.154995 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997"] Nov 02 20:15:23.155262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:23.155235 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" containerID="cri-o://c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2" gracePeriod=30 Nov 02 20:15:26.045456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:26.045413 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j"] Nov 02 20:15:26.091291 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:26.091256 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0deb60b3-cbw6j"] Nov 02 20:15:27.528480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:27.528441 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:15:27.528952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:27.528501 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:15:27.625824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:27.625791 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e" path="/var/lib/kubelet/pods/ba0722c5-b76a-4ad1-85aa-cabf1f2ca97e/volumes" Nov 02 20:15:27.890205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:27.890117 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" podUID="5495dc82-30ff-4357-94c8-3a077626f070" containerName="payload-processing" containerID="cri-o://913200d1f3193c51313405db8fb73a81ea9269ac671e9e5206604edb15f1272e" gracePeriod=25 Nov 02 20:15:27.992732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:27.992679 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" podUID="eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" containerName="payload-pre-processing" containerID="cri-o://01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604" gracePeriod=25 Nov 02 20:15:28.009514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.009487 5800 generic.go:356] "Generic (PLEG): container finished" podID="5495dc82-30ff-4357-94c8-3a077626f070" containerID="913200d1f3193c51313405db8fb73a81ea9269ac671e9e5206604edb15f1272e" exitCode=0 Nov 02 20:15:28.009602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.009556 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" event={"ID":"5495dc82-30ff-4357-94c8-3a077626f070","Type":"ContainerDied","Data":"913200d1f3193c51313405db8fb73a81ea9269ac671e9e5206604edb15f1272e"} Nov 02 20:15:28.332420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.328462 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:15:28.335368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.332742 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:15:28.417932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.417894 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5495dc82-30ff-4357-94c8-3a077626f070-kube-api-access-kcm7r\" (UniqueName: \"kubernetes.io/projected/5495dc82-30ff-4357-94c8-3a077626f070-kube-api-access-kcm7r\") pod \"5495dc82-30ff-4357-94c8-3a077626f070\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " Nov 02 20:15:28.418116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.418076 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5495dc82-30ff-4357-94c8-3a077626f070-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5495dc82-30ff-4357-94c8-3a077626f070-plugins-config-volume\") pod \"5495dc82-30ff-4357-94c8-3a077626f070\" (UID: \"5495dc82-30ff-4357-94c8-3a077626f070\") " Nov 02 20:15:28.418200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.418134 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-plugins-config-volume\") pod \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " Nov 02 20:15:28.418270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.418256 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-kube-api-access-gq9xk\" (UniqueName: \"kubernetes.io/projected/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-kube-api-access-gq9xk\") pod \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\" (UID: \"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4\") " Nov 02 20:15:28.419960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.419911 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-plugins-config-volume" pod "eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" (UID: "eed9aa4d-8cfc-4105-b17e-3f0a85d011a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:28.420080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.419975 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5495dc82-30ff-4357-94c8-3a077626f070-plugins-config-volume" pod "5495dc82-30ff-4357-94c8-3a077626f070" (UID: "5495dc82-30ff-4357-94c8-3a077626f070"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:28.426694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.425690 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-kube-api-access-gq9xk" pod "eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" (UID: "eed9aa4d-8cfc-4105-b17e-3f0a85d011a4"). InnerVolumeSpecName "kube-api-access-gq9xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:28.448352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.446184 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5495dc82-30ff-4357-94c8-3a077626f070-kube-api-access-kcm7r" pod "5495dc82-30ff-4357-94c8-3a077626f070" (UID: "5495dc82-30ff-4357-94c8-3a077626f070"). InnerVolumeSpecName "kube-api-access-kcm7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:28.473533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.473487 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:15:28.504874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.504843 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:15:28.520666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.520621 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9xk\" (UniqueName: \"kubernetes.io/projected/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-kube-api-access-gq9xk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.520666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.520658 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcm7r\" (UniqueName: \"kubernetes.io/projected/5495dc82-30ff-4357-94c8-3a077626f070-kube-api-access-kcm7r\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.520666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.520673 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5495dc82-30ff-4357-94c8-3a077626f070-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.520894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.520685 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.621321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621274 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-kube-api-access-j89jk\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-kube-api-access-j89jk\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621331 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-certs\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621375 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ecbc55f-3527-4934-b18f-278ccdcd7515-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecbc55f-3527-4934-b18f-278ccdcd7515-istiod-ca-cert\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621417 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-istio-token\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-istio-token\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621438 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-credential-socket\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621480 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-envoy\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621532 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-socket\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621574 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-envoy\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621587 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-certs" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621613 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-credential-socket\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621669 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-podinfo\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621697 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-kube-api-access-54582\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-kube-api-access-54582\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621737 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-data\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.621784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621778 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00022ea6-6d93-402b-8043-b8b1e013e185-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00022ea6-6d93-402b-8043-b8b1e013e185-istiod-ca-cert\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621819 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-socket" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621852 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00022ea6-6d93-402b-8043-b8b1e013e185-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00022ea6-6d93-402b-8043-b8b1e013e185-istio-podinfo\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621904 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-token\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-token\") pod \"5ecbc55f-3527-4934-b18f-278ccdcd7515\" (UID: \"5ecbc55f-3527-4934-b18f-278ccdcd7515\") " Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621933 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-data\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621929 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-credential-socket" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621953 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecbc55f-3527-4934-b18f-278ccdcd7515-istiod-ca-cert" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.621964 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-socket\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622022 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-credential-socket" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622152 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-certs\") pod \"00022ea6-6d93-402b-8043-b8b1e013e185\" (UID: \"00022ea6-6d93-402b-8043-b8b1e013e185\") " Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622163 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-socket" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.622484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622415 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00022ea6-6d93-402b-8043-b8b1e013e185-istiod-ca-cert" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622634 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-certs" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622932 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622955 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622970 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ecbc55f-3527-4934-b18f-278ccdcd7515-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622985 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.622999 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.623013 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.623027 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00022ea6-6d93-402b-8043-b8b1e013e185-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.623041 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.623525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.623440 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-data" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.624180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.624151 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-envoy" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.624326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.624250 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-kube-api-access-j89jk" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "kube-api-access-j89jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:28.624493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.624446 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-envoy" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.624569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.624541 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-data" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:28.624604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.624570 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00022ea6-6d93-402b-8043-b8b1e013e185-istio-podinfo" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:28.625014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.624990 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-token" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:28.625601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.625575 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-podinfo" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:28.625700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.625604 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-kube-api-access-54582" pod "5ecbc55f-3527-4934-b18f-278ccdcd7515" (UID: "5ecbc55f-3527-4934-b18f-278ccdcd7515"). InnerVolumeSpecName "kube-api-access-54582". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:28.625964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.625934 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-istio-token" pod "00022ea6-6d93-402b-8043-b8b1e013e185" (UID: "00022ea6-6d93-402b-8043-b8b1e013e185"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:28.724791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724707 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.724791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724735 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.724791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724748 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.724791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724764 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54582\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-kube-api-access-54582\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.724791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724778 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.724791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724790 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00022ea6-6d93-402b-8043-b8b1e013e185-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.725093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724803 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ecbc55f-3527-4934-b18f-278ccdcd7515-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.725093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724814 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00022ea6-6d93-402b-8043-b8b1e013e185-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.725093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724826 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j89jk\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-kube-api-access-j89jk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:28.725093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:28.724837 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00022ea6-6d93-402b-8043-b8b1e013e185-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:29.018508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.018479 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" Nov 02 20:15:29.018698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.018505 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf" event={"ID":"5495dc82-30ff-4357-94c8-3a077626f070","Type":"ContainerDied","Data":"5e9859a1bc47669bc60b228f57bd188d0a2de4a4487c5552fe9a5f658db4a327"} Nov 02 20:15:29.018698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.018561 5800 scope.go:122] "RemoveContainer" containerID="913200d1f3193c51313405db8fb73a81ea9269ac671e9e5206604edb15f1272e" Nov 02 20:15:29.020112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.020084 5800 generic.go:356] "Generic (PLEG): container finished" podID="00022ea6-6d93-402b-8043-b8b1e013e185" containerID="c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2" exitCode=0 Nov 02 20:15:29.020221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.020143 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" Nov 02 20:15:29.020221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.020165 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" event={"ID":"00022ea6-6d93-402b-8043-b8b1e013e185","Type":"ContainerDied","Data":"c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2"} Nov 02 20:15:29.020348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.020216 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997" event={"ID":"00022ea6-6d93-402b-8043-b8b1e013e185","Type":"ContainerDied","Data":"1f39c563d9c10c44bf8fc35bbcfda9ab3776ddba2c352b9ff9383864b6fe9415"} Nov 02 20:15:29.021621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.021599 5800 generic.go:356] "Generic (PLEG): container finished" podID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerID="3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1" exitCode=0 Nov 02 20:15:29.021718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.021639 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" Nov 02 20:15:29.021718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.021680 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" event={"ID":"5ecbc55f-3527-4934-b18f-278ccdcd7515","Type":"ContainerDied","Data":"3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1"} Nov 02 20:15:29.021833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.021727 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4" event={"ID":"5ecbc55f-3527-4934-b18f-278ccdcd7515","Type":"ContainerDied","Data":"5bdd8b81aa5f37ad9ce1bc4c2f3914508acbadea204a3262c6f68ab2d4abb125"} Nov 02 20:15:29.023336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.023313 5800 generic.go:356] "Generic (PLEG): container finished" podID="eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" containerID="01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604" exitCode=0 Nov 02 20:15:29.023434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.023344 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" event={"ID":"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4","Type":"ContainerDied","Data":"01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604"} Nov 02 20:15:29.023434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.023373 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" event={"ID":"eed9aa4d-8cfc-4105-b17e-3f0a85d011a4","Type":"ContainerDied","Data":"cc6e0cf849ecea9df91f02899ba3ac954390a140b4660fa284c90833f5bc313d"} Nov 02 20:15:29.023434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.023392 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq" Nov 02 20:15:29.032626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.032604 5800 scope.go:122] "RemoveContainer" containerID="c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2" Nov 02 20:15:29.044731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.044714 5800 scope.go:122] "RemoveContainer" containerID="c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2" Nov 02 20:15:29.045043 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:29.045023 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2\": container with ID starting with c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2 not found: ID does not exist" containerID="c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2" Nov 02 20:15:29.045103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.045051 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2"} err="failed to get container status \"c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2\": rpc error: code = NotFound desc = could not find container \"c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2\": container with ID starting with c07ea5ba71c34604cb7fa8806f44f107d6677af984ab024ddcc0c226b3bc74d2 not found: ID does not exist" Nov 02 20:15:29.045103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.045069 5800 scope.go:122] "RemoveContainer" containerID="3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1" Nov 02 20:15:29.055238 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.055217 5800 scope.go:122] "RemoveContainer" containerID="3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1" Nov 02 20:15:29.055552 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:29.055533 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1\": container with ID starting with 3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1 not found: ID does not exist" containerID="3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1" Nov 02 20:15:29.055627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.055557 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1"} err="failed to get container status \"3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1\": rpc error: code = NotFound desc = could not find container \"3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1\": container with ID starting with 3b2ed62015bb1e3bb17f5ed27c33e62cd6f84532a5670e0b2e35dd0b3bdfe8c1 not found: ID does not exist" Nov 02 20:15:29.055627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.055575 5800 scope.go:122] "RemoveContainer" containerID="01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604" Nov 02 20:15:29.064657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.064639 5800 scope.go:122] "RemoveContainer" containerID="01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604" Nov 02 20:15:29.064919 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:29.064901 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604\": container with ID starting with 01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604 not found: ID does not exist" containerID="01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604" Nov 02 20:15:29.064990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.064925 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604"} err="failed to get container status \"01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604\": rpc error: code = NotFound desc = could not find container \"01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604\": container with ID starting with 01c7e8fbf2f853da0f0fd5bdd1c30a62a452722219dcf4a3a1c42d9de3070604 not found: ID does not exist" Nov 02 20:15:29.174668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.174638 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4"] Nov 02 20:15:29.178221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.178191 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-4q4l4"] Nov 02 20:15:29.278691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.278611 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf"] Nov 02 20:15:29.284247 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.284221 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-341fb2-5c6bf7f6d5-chlvf"] Nov 02 20:15:29.385298 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.385255 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq"] Nov 02 20:15:29.390470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.390441 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-341fb2-77c748876c-cqdtq"] Nov 02 20:15:29.492020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.491989 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997"] Nov 02 20:15:29.495401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.495380 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-m2997"] Nov 02 20:15:29.624945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.624870 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" path="/var/lib/kubelet/pods/00022ea6-6d93-402b-8043-b8b1e013e185/volumes" Nov 02 20:15:29.625368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.625351 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5495dc82-30ff-4357-94c8-3a077626f070" path="/var/lib/kubelet/pods/5495dc82-30ff-4357-94c8-3a077626f070/volumes" Nov 02 20:15:29.625809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.625796 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" path="/var/lib/kubelet/pods/5ecbc55f-3527-4934-b18f-278ccdcd7515/volumes" Nov 02 20:15:29.626565 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:29.626548 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" path="/var/lib/kubelet/pods/eed9aa4d-8cfc-4105-b17e-3f0a85d011a4/volumes" Nov 02 20:15:32.549470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:32.549424 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn"] Nov 02 20:15:36.745770 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:36.745730 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27"] Nov 02 20:15:36.746192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:36.746041 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" containerID="cri-o://89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd" gracePeriod=30 Nov 02 20:15:37.550806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:37.550758 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" podUID="f0971513-3f2f-49b1-814b-23362b80e9d0" containerName="payload-pre-processing" containerID="cri-o://41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d" gracePeriod=25 Nov 02 20:15:37.888438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:37.888407 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:15:37.920979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:37.920947 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0971513-3f2f-49b1-814b-23362b80e9d0-kube-api-access-n7kss\" (UniqueName: \"kubernetes.io/projected/f0971513-3f2f-49b1-814b-23362b80e9d0-kube-api-access-n7kss\") pod \"f0971513-3f2f-49b1-814b-23362b80e9d0\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " Nov 02 20:15:37.921167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:37.921093 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0971513-3f2f-49b1-814b-23362b80e9d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0971513-3f2f-49b1-814b-23362b80e9d0-plugins-config-volume\") pod \"f0971513-3f2f-49b1-814b-23362b80e9d0\" (UID: \"f0971513-3f2f-49b1-814b-23362b80e9d0\") " Nov 02 20:15:37.921492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:37.921467 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0971513-3f2f-49b1-814b-23362b80e9d0-plugins-config-volume" pod "f0971513-3f2f-49b1-814b-23362b80e9d0" (UID: "f0971513-3f2f-49b1-814b-23362b80e9d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:37.921603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:37.921579 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0971513-3f2f-49b1-814b-23362b80e9d0-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:37.923146 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:37.923114 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0971513-3f2f-49b1-814b-23362b80e9d0-kube-api-access-n7kss" pod "f0971513-3f2f-49b1-814b-23362b80e9d0" (UID: "f0971513-3f2f-49b1-814b-23362b80e9d0"). InnerVolumeSpecName "kube-api-access-n7kss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:38.023664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.023614 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7kss\" (UniqueName: \"kubernetes.io/projected/f0971513-3f2f-49b1-814b-23362b80e9d0-kube-api-access-n7kss\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:38.107657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.107624 5800 generic.go:356] "Generic (PLEG): container finished" podID="f0971513-3f2f-49b1-814b-23362b80e9d0" containerID="41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d" exitCode=0 Nov 02 20:15:38.107806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.107671 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" Nov 02 20:15:38.107806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.107688 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" event={"ID":"f0971513-3f2f-49b1-814b-23362b80e9d0","Type":"ContainerDied","Data":"41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d"} Nov 02 20:15:38.107806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.107734 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn" event={"ID":"f0971513-3f2f-49b1-814b-23362b80e9d0","Type":"ContainerDied","Data":"8de4033199e3603493660a559073760e911074a810205627c0c95934465caa39"} Nov 02 20:15:38.107806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.107755 5800 scope.go:122] "RemoveContainer" containerID="41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d" Nov 02 20:15:38.119354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.119335 5800 scope.go:122] "RemoveContainer" containerID="41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d" Nov 02 20:15:38.119637 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:38.119618 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d\": container with ID starting with 41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d not found: ID does not exist" containerID="41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d" Nov 02 20:15:38.119714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.119645 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d"} err="failed to get container status \"41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d\": rpc error: code = NotFound desc = could not find container \"41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d\": container with ID starting with 41f2ddc9bb02f3653e5e63832e612b455bbc2055577721cef7c85dbe7586d71d not found: ID does not exist" Nov 02 20:15:38.264444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.264408 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn"] Nov 02 20:15:38.273980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.273951 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3021d-5b49657d75-vlxtn"] Nov 02 20:15:38.520347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.520309 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg"] Nov 02 20:15:38.520561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.520539 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" containerID="cri-o://bdf2786840974519087e29e6ae2419a3cabd81945085548902d6439a2a5c1214" gracePeriod=30 Nov 02 20:15:38.524713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.524684 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh"] Nov 02 20:15:38.525011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.524953 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" containerID="cri-o://1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f" gracePeriod=30 Nov 02 20:15:38.534695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.534358 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv"] Nov 02 20:15:38.534695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.534628 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" containerID="cri-o://c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df" gracePeriod=30 Nov 02 20:15:38.539414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.539381 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9"] Nov 02 20:15:38.539649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:38.539623 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" containerID="cri-o://e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427" gracePeriod=30 Nov 02 20:15:39.624010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:39.623963 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0971513-3f2f-49b1-814b-23362b80e9d0" path="/var/lib/kubelet/pods/f0971513-3f2f-49b1-814b-23362b80e9d0/volumes" Nov 02 20:15:42.021246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.021186 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:15:42.086439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086407 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-socket\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086443 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-token\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-token\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086574 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-kube-api-access-gpvts\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-kube-api-access-gpvts\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086610 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-credential-socket\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086678 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-podinfo\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086699 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-socket" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:42.086779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086718 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-envoy\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086765 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-certs\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086852 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istiod-ca-cert\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086964 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-data\") pod \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\" (UID: \"099c86c1-d83a-450d-8ae0-908bac2cc0d4\") " Nov 02 20:15:42.086981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.086967 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-credential-socket" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:42.087157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087136 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-certs" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:42.087382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087357 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-data" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:42.087477 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087404 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istiod-ca-cert" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:42.087715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087691 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.087833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087720 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.087833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087737 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.087833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087751 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.087833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.087767 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.088969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.088943 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-podinfo" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:42.089059 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.088995 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-token" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:42.089213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.089191 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-kube-api-access-gpvts" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "kube-api-access-gpvts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:42.089361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.089342 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-envoy" pod "099c86c1-d83a-450d-8ae0-908bac2cc0d4" (UID: "099c86c1-d83a-450d-8ae0-908bac2cc0d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:42.139355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.139321 5800 generic.go:356] "Generic (PLEG): container finished" podID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerID="89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd" exitCode=0 Nov 02 20:15:42.139508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.139384 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" Nov 02 20:15:42.139508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.139412 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" event={"ID":"099c86c1-d83a-450d-8ae0-908bac2cc0d4","Type":"ContainerDied","Data":"89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd"} Nov 02 20:15:42.139508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.139487 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27" event={"ID":"099c86c1-d83a-450d-8ae0-908bac2cc0d4","Type":"ContainerDied","Data":"525f20bec04542176ea28575491901e9b4e290f44144a90a922984f1ed16f84f"} Nov 02 20:15:42.139701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.139523 5800 scope.go:122] "RemoveContainer" containerID="89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd" Nov 02 20:15:42.151540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.151520 5800 scope.go:122] "RemoveContainer" containerID="89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd" Nov 02 20:15:42.151818 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:42.151796 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd\": container with ID starting with 89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd not found: ID does not exist" containerID="89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd" Nov 02 20:15:42.151894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.151824 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd"} err="failed to get container status \"89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd\": rpc error: code = NotFound desc = could not find container \"89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd\": container with ID starting with 89d4a5685523d072989b0333b26e6acaf3b48fde87e93cc0c3df2b30c81e28fd not found: ID does not exist" Nov 02 20:15:42.189136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.189105 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.189136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.189137 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.189136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.189152 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.189391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.189166 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvts\" (UniqueName: \"kubernetes.io/projected/099c86c1-d83a-450d-8ae0-908bac2cc0d4-kube-api-access-gpvts\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:42.283748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.283677 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27"] Nov 02 20:15:42.289569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:42.289536 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-g7f27"] Nov 02 20:15:43.624486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.624455 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" path="/var/lib/kubelet/pods/099c86c1-d83a-450d-8ae0-908bac2cc0d4/volumes" Nov 02 20:15:43.948733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.948694 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:15:43.969961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.969925 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l"] Nov 02 20:15:43.970907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970882 5800 cpu_manager.go:414] "containerMap: removing container" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:43.970907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970904 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970928 5800 cpu_manager.go:414] "containerMap: removing container" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970936 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970951 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970960 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970971 5800 cpu_manager.go:414] "containerMap: removing container" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970979 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.970992 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971000 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971022 5800 cpu_manager.go:414] "containerMap: removing container" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971029 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971053 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971061 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971073 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971080 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971089 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerName="cleanup" Nov 02 20:15:43.971103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971097 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971114 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f0971513-3f2f-49b1-814b-23362b80e9d0" containerName="payload-pre-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971122 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0971513-3f2f-49b1-814b-23362b80e9d0" containerName="payload-pre-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971135 5800 cpu_manager.go:414] "containerMap: removing container" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971142 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971156 5800 cpu_manager.go:414] "containerMap: removing container" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971163 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971177 5800 cpu_manager.go:414] "containerMap: removing container" podUID="eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" containerName="payload-pre-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971185 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" containerName="payload-pre-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971198 5800 cpu_manager.go:414] "containerMap: removing container" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971206 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971220 5800 cpu_manager.go:414] "containerMap: removing container" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971227 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971240 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971249 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971265 5800 cpu_manager.go:414] "containerMap: removing container" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971272 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971298 5800 cpu_manager.go:414] "containerMap: removing container" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971305 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971319 5800 cpu_manager.go:414] "containerMap: removing container" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971326 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971342 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5495dc82-30ff-4357-94c8-3a077626f070" containerName="payload-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971349 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5495dc82-30ff-4357-94c8-3a077626f070" containerName="payload-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971373 5800 cpu_manager.go:414] "containerMap: removing container" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971380 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971390 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971397 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971541 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971553 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971560 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971570 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971581 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971589 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971600 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971608 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971618 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971629 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971641 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971649 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ecbc55f-3527-4934-b18f-278ccdcd7515" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971657 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba91842-792f-45bc-831a-60dd4419cdcd" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971666 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0971513-3f2f-49b1-814b-23362b80e9d0" containerName="payload-pre-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971675 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971684 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971691 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed9aa4d-8cfc-4105-b17e-3f0a85d011a4" containerName="payload-pre-processing" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971702 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="099c86c1-d83a-450d-8ae0-908bac2cc0d4" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971710 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971720 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26e053c-6975-4224-ac62-9a054e216d84" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971729 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="00022ea6-6d93-402b-8043-b8b1e013e185" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971738 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="987e1d4c-33df-4a9d-8f31-6a292c2c8fff" containerName="cleanup" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971747 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e31a97-85a6-4372-9d49-9a59d5da22b4" containerName="istio-proxy" Nov 02 20:15:43.972033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.971757 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5495dc82-30ff-4357-94c8-3a077626f070" containerName="payload-processing" Nov 02 20:15:43.975908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:43.975889 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.010647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.010622 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:15:44.014461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.014440 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:15:44.018359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.017252 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d326b4-4235-4f45-a9f2-be2f5561d609-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.018359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.017441 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wm5\" (UniqueName: \"kubernetes.io/projected/b1d326b4-4235-4f45-a9f2-be2f5561d609-kube-api-access-52wm5\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.024617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.024590 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l"] Nov 02 20:15:44.044245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.044079 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5e04421b-dockercfg-5mmfd\"" Nov 02 20:15:44.044245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.044163 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5e04421b\"" Nov 02 20:15:44.147473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.128259 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-envoy\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.147732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.147706 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-certs\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.147832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.147823 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-certs\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.147908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.147900 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-envoy\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.147982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.147974 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-data\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.148070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148061 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-kube-api-access-hg54n\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-kube-api-access-hg54n\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.148154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148145 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-credential-socket\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.148255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148247 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-kube-api-access-rwsph\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-kube-api-access-rwsph\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.148392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148370 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-credential-socket\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.148530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148430 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-podinfo\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.148530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148469 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-data\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.148530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148512 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istiod-ca-cert\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.148695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148557 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-data\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.148695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148595 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-token\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-token\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.148695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148639 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-podinfo\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.148695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148688 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-token\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-token\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.148892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148728 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-socket\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.148892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148767 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-credential-socket\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.148892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148828 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-kube-api-access-t8h9z\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-kube-api-access-t8h9z\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.148892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148880 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istiod-ca-cert\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.149101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148920 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-socket\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.149101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148960 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-envoy\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.149101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.148999 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istiod-ca-cert\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.149101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.149045 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-socket\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.149311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.149104 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-token\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-token\") pod \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\" (UID: \"4448bd32-9a2a-4cfd-9dd2-ee343df34df5\") " Nov 02 20:15:44.149311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.149160 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-podinfo\") pod \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\" (UID: \"ed74e3bc-2332-47fe-9276-6e4aa8161f14\") " Nov 02 20:15:44.149311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.149228 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-certs\") pod \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\" (UID: \"3de6c98b-c7c5-452c-ba6c-87179d8a5127\") " Nov 02 20:15:44.149676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.149652 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d326b4-4235-4f45-a9f2-be2f5561d609-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.149853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.149839 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wm5\" (UniqueName: \"kubernetes.io/projected/b1d326b4-4235-4f45-a9f2-be2f5561d609-kube-api-access-52wm5\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.152721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.152693 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-envoy" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.153082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.153063 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-certs" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.153377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.153359 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-certs" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.160786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.160536 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istiod-ca-cert" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:44.161849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.161566 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-certs" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.166353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.164168 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-socket" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.172130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.170263 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d326b4-4235-4f45-a9f2-be2f5561d609-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.172130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.171312 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-data" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.172130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.171494 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istiod-ca-cert" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:44.172130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.171859 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istiod-ca-cert" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:44.172564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.172523 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-data" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.173387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.172731 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-credential-socket" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.173387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.172996 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-socket" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.173387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.173200 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-credential-socket" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.174038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.173775 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-credential-socket" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.174634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.174597 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-socket" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.174823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.174798 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-data" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.183382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.181416 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-kube-api-access-t8h9z" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "kube-api-access-t8h9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.185140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.185110 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-token" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.185900 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.185851 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-kube-api-access-hg54n" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "kube-api-access-hg54n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.186338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.186314 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-token" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.186661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.186623 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-kube-api-access-rwsph" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "kube-api-access-rwsph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.187181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.187147 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-podinfo" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:44.190836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.190259 5800 generic.go:356] "Generic (PLEG): container finished" podID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerID="e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427" exitCode=0 Nov 02 20:15:44.190836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.190380 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" event={"ID":"3de6c98b-c7c5-452c-ba6c-87179d8a5127","Type":"ContainerDied","Data":"e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427"} Nov 02 20:15:44.190836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.190433 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" event={"ID":"3de6c98b-c7c5-452c-ba6c-87179d8a5127","Type":"ContainerDied","Data":"27ca4d2458a1f10430bd67f8fb16514c37931892924b882ef7e4dca74de7dd72"} Nov 02 20:15:44.190836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.190462 5800 scope.go:122] "RemoveContainer" containerID="e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427" Nov 02 20:15:44.190836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.190623 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9" Nov 02 20:15:44.197504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.197216 5800 generic.go:356] "Generic (PLEG): container finished" podID="200795e0-806f-4df7-98d2-a3594072c700" containerID="bdf2786840974519087e29e6ae2419a3cabd81945085548902d6439a2a5c1214" exitCode=0 Nov 02 20:15:44.197504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.197424 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" event={"ID":"200795e0-806f-4df7-98d2-a3594072c700","Type":"ContainerDied","Data":"bdf2786840974519087e29e6ae2419a3cabd81945085548902d6439a2a5c1214"} Nov 02 20:15:44.199994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.199781 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-envoy" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.200199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.200169 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-podinfo" pod "4448bd32-9a2a-4cfd-9dd2-ee343df34df5" (UID: "4448bd32-9a2a-4cfd-9dd2-ee343df34df5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:44.202519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.201561 5800 generic.go:356] "Generic (PLEG): container finished" podID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" containerID="1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f" exitCode=0 Nov 02 20:15:44.202519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.201696 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" event={"ID":"ed74e3bc-2332-47fe-9276-6e4aa8161f14","Type":"ContainerDied","Data":"1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f"} Nov 02 20:15:44.202519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.201743 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" event={"ID":"ed74e3bc-2332-47fe-9276-6e4aa8161f14","Type":"ContainerDied","Data":"eab75f6dda9e85a2796acdf77ecb8f65cf5c53e5ede2f5526d598855acc333b1"} Nov 02 20:15:44.202519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.201833 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh" Nov 02 20:15:44.203930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.203814 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-token" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.204311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.204273 5800 generic.go:356] "Generic (PLEG): container finished" podID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerID="c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df" exitCode=0 Nov 02 20:15:44.204548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.204526 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" event={"ID":"4448bd32-9a2a-4cfd-9dd2-ee343df34df5","Type":"ContainerDied","Data":"c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df"} Nov 02 20:15:44.204707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.204688 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" event={"ID":"4448bd32-9a2a-4cfd-9dd2-ee343df34df5","Type":"ContainerDied","Data":"15b941693c19393dbe333e7e9011d670a221a24fcc146f9f987ed12bd8fd686c"} Nov 02 20:15:44.205257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.205238 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv" Nov 02 20:15:44.207236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.207214 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-envoy" pod "3de6c98b-c7c5-452c-ba6c-87179d8a5127" (UID: "3de6c98b-c7c5-452c-ba6c-87179d8a5127"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.212502 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs"] Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213412 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213429 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213454 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213462 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213480 5800 cpu_manager.go:414] "containerMap: removing container" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213488 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606b6ef9-c443-4760-bf82-462f49e08d23" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213500 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213508 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213522 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213528 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213549 5800 cpu_manager.go:414] "containerMap: removing container" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213555 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189e8ce6-3ae1-43e0-a637-0300c3f9deff" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213583 5800 cpu_manager.go:414] "containerMap: removing container" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213591 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213733 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213752 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="59041d79-d3f5-42d6-bf89-633f228cd747" containerName="cleanup" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213766 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" containerName="istio-proxy" Nov 02 20:15:44.215622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.213775 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d920b9b6-8c49-4320-9ca8-fbeb37c75703" containerName="cleanup" Nov 02 20:15:44.218009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.217983 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-podinfo" pod "ed74e3bc-2332-47fe-9276-6e4aa8161f14" (UID: "ed74e3bc-2332-47fe-9276-6e4aa8161f14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:44.222538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.220797 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.227729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.227696 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs"] Nov 02 20:15:44.236040 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.236005 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wm5\" (UniqueName: \"kubernetes.io/projected/b1d326b4-4235-4f45-a9f2-be2f5561d609-kube-api-access-52wm5\") pod \"payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.258000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.257961 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqqc\" (UniqueName: \"kubernetes.io/projected/5ddd6607-38d2-4d08-ada6-3138fcdc84db-kube-api-access-wqqqc\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258333 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddd6607-38d2-4d08-ada6-3138fcdc84db-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258499 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8h9z\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-kube-api-access-t8h9z\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258519 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258533 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258546 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258560 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258574 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258587 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258601 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258615 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258628 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258641 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258656 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258669 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258682 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258695 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg54n\" (UniqueName: \"kubernetes.io/projected/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-kube-api-access-hg54n\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258708 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258722 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwsph\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-kube-api-access-rwsph\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258735 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258749 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258762 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258776 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258791 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258805 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258819 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de6c98b-c7c5-452c-ba6c-87179d8a5127-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258833 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed74e3bc-2332-47fe-9276-6e4aa8161f14-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258846 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4448bd32-9a2a-4cfd-9dd2-ee343df34df5-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.258909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.258861 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de6c98b-c7c5-452c-ba6c-87179d8a5127-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.261556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.261448 5800 scope.go:122] "RemoveContainer" containerID="e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12" Nov 02 20:15:44.282746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.282680 5800 scope.go:122] "RemoveContainer" containerID="e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427" Nov 02 20:15:44.283273 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:44.283242 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427\": container with ID starting with e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427 not found: ID does not exist" containerID="e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427" Nov 02 20:15:44.283273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.283308 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427"} err="failed to get container status \"e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427\": rpc error: code = NotFound desc = could not find container \"e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427\": container with ID starting with e50d841f8708f89b0b219dfda4a881b2de51fb2da88ff21553b3d1b40c644427 not found: ID does not exist" Nov 02 20:15:44.283273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.283337 5800 scope.go:122] "RemoveContainer" containerID="e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12" Nov 02 20:15:44.283273 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:44.283629 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12\": container with ID starting with e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12 not found: ID does not exist" containerID="e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12" Nov 02 20:15:44.283273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.283658 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12"} err="failed to get container status \"e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12\": rpc error: code = NotFound desc = could not find container \"e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12\": container with ID starting with e3a4edc28ee1874e3edddeaf7c2cfafaffa0d2a3ffada5eebf037bd7bda62d12 not found: ID does not exist" Nov 02 20:15:44.283273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.283676 5800 scope.go:122] "RemoveContainer" containerID="1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f" Nov 02 20:15:44.292852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.292642 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:44.302726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.302704 5800 scope.go:122] "RemoveContainer" containerID="1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f" Nov 02 20:15:44.303062 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:44.303033 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f\": container with ID starting with 1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f not found: ID does not exist" containerID="1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f" Nov 02 20:15:44.303157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.303074 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f"} err="failed to get container status \"1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f\": rpc error: code = NotFound desc = could not find container \"1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f\": container with ID starting with 1eff6d8c0dfde7d30befa7a1d157799780034702ff23d697efdcb1f3ae83a28f not found: ID does not exist" Nov 02 20:15:44.303157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.303097 5800 scope.go:122] "RemoveContainer" containerID="c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df" Nov 02 20:15:44.317784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.317763 5800 scope.go:122] "RemoveContainer" containerID="c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df" Nov 02 20:15:44.318111 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:44.318090 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df\": container with ID starting with c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df not found: ID does not exist" containerID="c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df" Nov 02 20:15:44.318206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.318123 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df"} err="failed to get container status \"c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df\": rpc error: code = NotFound desc = could not find container \"c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df\": container with ID starting with c0f8bdcf323392bacab2539010df216042f40049b625bcbe87fb9190a95fb9df not found: ID does not exist" Nov 02 20:15:44.372354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.364269 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddd6607-38d2-4d08-ada6-3138fcdc84db-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.372354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.364462 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqqc\" (UniqueName: \"kubernetes.io/projected/5ddd6607-38d2-4d08-ada6-3138fcdc84db-kube-api-access-wqqqc\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.372354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.365777 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddd6607-38d2-4d08-ada6-3138fcdc84db-plugins-config-volume\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.417912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.417848 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqqc\" (UniqueName: \"kubernetes.io/projected/5ddd6607-38d2-4d08-ada6-3138fcdc84db-kube-api-access-wqqqc\") pod \"payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.420575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.420535 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:15:44.465654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.465614 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200795e0-806f-4df7-98d2-a3594072c700-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200795e0-806f-4df7-98d2-a3594072c700-istiod-ca-cert\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.465820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.465742 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-certs\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.465905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.465880 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-istio-token\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-istio-token\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.465969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.465954 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/200795e0-806f-4df7-98d2-a3594072c700-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200795e0-806f-4df7-98d2-a3594072c700-istio-podinfo\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.466058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.465980 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-data\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.466058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466026 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-socket\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.466168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466121 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-envoy\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.466168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466140 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-certs" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.467393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466163 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-credential-socket\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.467393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466205 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-kube-api-access-nrt2g\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-kube-api-access-nrt2g\") pod \"200795e0-806f-4df7-98d2-a3594072c700\" (UID: \"200795e0-806f-4df7-98d2-a3594072c700\") " Nov 02 20:15:44.467393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466360 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-data" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.467393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466797 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.467393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.466818 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.467701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.467503 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-credential-socket" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.467701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.467561 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-socket" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.467788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.467714 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200795e0-806f-4df7-98d2-a3594072c700-istiod-ca-cert" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:15:44.469578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.469536 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/200795e0-806f-4df7-98d2-a3594072c700-istio-podinfo" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:15:44.469783 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.469762 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-envoy" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:15:44.470195 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.470170 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-kube-api-access-nrt2g" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "kube-api-access-nrt2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.471675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.471639 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-istio-token" pod "200795e0-806f-4df7-98d2-a3594072c700" (UID: "200795e0-806f-4df7-98d2-a3594072c700"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:15:44.567821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.567789 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200795e0-806f-4df7-98d2-a3594072c700-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.567821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.567818 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.567821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.567827 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.568096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.567836 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200795e0-806f-4df7-98d2-a3594072c700-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.568096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.567846 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrt2g\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-kube-api-access-nrt2g\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.568096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.567856 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200795e0-806f-4df7-98d2-a3594072c700-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.568096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.567866 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200795e0-806f-4df7-98d2-a3594072c700-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:15:44.574469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.574440 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:44.692960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.692838 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv"] Nov 02 20:15:44.698342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.698313 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-g4ngv"] Nov 02 20:15:44.729860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.729575 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l"] Nov 02 20:15:44.736842 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:44.736808 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d326b4_4235_4f45_a9f2_be2f5561d609.slice/crio-cd9e6a58f0dbce57e02b0c62512fecb62bcf29806e06e99e7921728b1e7b85b6 WatchSource:0}: Error finding container cd9e6a58f0dbce57e02b0c62512fecb62bcf29806e06e99e7921728b1e7b85b6: Status 404 returned error can't find the container with id cd9e6a58f0dbce57e02b0c62512fecb62bcf29806e06e99e7921728b1e7b85b6 Nov 02 20:15:44.946304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.946207 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh"] Nov 02 20:15:44.959199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.959159 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-2wzsh"] Nov 02 20:15:44.959871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:44.959853 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs"] Nov 02 20:15:44.961959 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:44.961930 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddd6607_38d2_4d08_ada6_3138fcdc84db.slice/crio-bb096e4ce2c1bcd1f2d75a6ef1aeb5d2848212ad3a01795f24b5d94804239bf0 WatchSource:0}: Error finding container bb096e4ce2c1bcd1f2d75a6ef1aeb5d2848212ad3a01795f24b5d94804239bf0: Status 404 returned error can't find the container with id bb096e4ce2c1bcd1f2d75a6ef1aeb5d2848212ad3a01795f24b5d94804239bf0 Nov 02 20:15:45.054743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.054687 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9"] Nov 02 20:15:45.058173 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.058151 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-cjcc9"] Nov 02 20:15:45.216938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.216837 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" Nov 02 20:15:45.216938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.216846 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg" event={"ID":"200795e0-806f-4df7-98d2-a3594072c700","Type":"ContainerDied","Data":"29317abbc40f36d212a0fccaf6dfe7f2b22257e88701d33a770d06c152879c4d"} Nov 02 20:15:45.217201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.217004 5800 scope.go:122] "RemoveContainer" containerID="bdf2786840974519087e29e6ae2419a3cabd81945085548902d6439a2a5c1214" Nov 02 20:15:45.218928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.218905 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" event={"ID":"b1d326b4-4235-4f45-a9f2-be2f5561d609","Type":"ContainerStarted","Data":"4a863d309587958a7f481b5e0e9a0bb0cf5bd2c01341e379f4170b0de248d87d"} Nov 02 20:15:45.219057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.218959 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:45.219057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.218985 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" event={"ID":"b1d326b4-4235-4f45-a9f2-be2f5561d609","Type":"ContainerStarted","Data":"cd9e6a58f0dbce57e02b0c62512fecb62bcf29806e06e99e7921728b1e7b85b6"} Nov 02 20:15:45.220746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.220720 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" event={"ID":"5ddd6607-38d2-4d08-ada6-3138fcdc84db","Type":"ContainerStarted","Data":"3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a"} Nov 02 20:15:45.220839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.220771 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" event={"ID":"5ddd6607-38d2-4d08-ada6-3138fcdc84db","Type":"ContainerStarted","Data":"bb096e4ce2c1bcd1f2d75a6ef1aeb5d2848212ad3a01795f24b5d94804239bf0"} Nov 02 20:15:45.220839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.220802 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:45.380580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.380528 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" podStartSLOduration=1.380515151 podStartE2EDuration="1.380515151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:45.378406001 +0000 UTC m=+1643.895178588" watchObservedRunningTime="2026-11-02 20:15:45.380515151 +0000 UTC m=+1643.897287792" Nov 02 20:15:45.623804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.623770 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" path="/var/lib/kubelet/pods/3de6c98b-c7c5-452c-ba6c-87179d8a5127/volumes" Nov 02 20:15:45.624273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.624260 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4448bd32-9a2a-4cfd-9dd2-ee343df34df5" path="/var/lib/kubelet/pods/4448bd32-9a2a-4cfd-9dd2-ee343df34df5/volumes" Nov 02 20:15:45.624740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.624721 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed74e3bc-2332-47fe-9276-6e4aa8161f14" path="/var/lib/kubelet/pods/ed74e3bc-2332-47fe-9276-6e4aa8161f14/volumes" Nov 02 20:15:45.640469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.640393 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" podStartSLOduration=2.64037116 podStartE2EDuration="2.64037116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:45.53020105 +0000 UTC m=+1644.046973635" watchObservedRunningTime="2026-11-02 20:15:45.64037116 +0000 UTC m=+1644.157143741" Nov 02 20:15:45.644583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.644549 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg"] Nov 02 20:15:45.649846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:45.649803 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-89rrg"] Nov 02 20:15:47.624072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:47.624039 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200795e0-806f-4df7-98d2-a3594072c700" path="/var/lib/kubelet/pods/200795e0-806f-4df7-98d2-a3594072c700/volumes" Nov 02 20:15:51.735013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.734974 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt"] Nov 02 20:15:51.735946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.735922 5800 cpu_manager.go:414] "containerMap: removing container" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" Nov 02 20:15:51.735991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.735949 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" Nov 02 20:15:51.736175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.736163 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="200795e0-806f-4df7-98d2-a3594072c700" containerName="istio-proxy" Nov 02 20:15:51.736211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.736182 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de6c98b-c7c5-452c-ba6c-87179d8a5127" containerName="istio-proxy" Nov 02 20:15:51.742098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.742066 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.801301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.797870 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-40aecd-openshift-default-dockercfg-sm8fz\"" Nov 02 20:15:51.910447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.910406 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt"] Nov 02 20:15:51.913576 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913543 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbr72\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-kube-api-access-gbr72\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.913744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913587 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.913744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913628 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.913744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913690 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.913744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913731 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.914038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913760 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.914038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913781 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.914038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.913981 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:51.914038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:51.914027 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585fcd70-8a96-4a96-88b7-f088fad23b01-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016123 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbr72\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-kube-api-access-gbr72\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016183 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016238 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016355 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016418 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016468 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016499 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016559 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016595 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585fcd70-8a96-4a96-88b7-f088fad23b01-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016871 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.016965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016928 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.017012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.016935 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.017056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.017023 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.017435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.017413 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585fcd70-8a96-4a96-88b7-f088fad23b01-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.018965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.018936 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.024337 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.019545 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.069324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.069274 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.069505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.069408 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbr72\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-kube-api-access-gbr72\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.360155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.360055 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:52.698056 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:15:52.697964 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:15:52.784976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.784947 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt"] Nov 02 20:15:52.785360 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:15:52.785076 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585fcd70_8a96_4a96_88b7_f088fad23b01.slice/crio-2acf0120825f6513623a56b0a7952d58e7917d57653d5d09efb0683231f7b424 WatchSource:0}: Error finding container 2acf0120825f6513623a56b0a7952d58e7917d57653d5d09efb0683231f7b424: Status 404 returned error can't find the container with id 2acf0120825f6513623a56b0a7952d58e7917d57653d5d09efb0683231f7b424 Nov 02 20:15:52.787714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.787682 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:52.787815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.787750 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:52.787815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:52.787778 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:15:53.287679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:53.287638 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" event={"ID":"585fcd70-8a96-4a96-88b7-f088fad23b01","Type":"ContainerStarted","Data":"7d25e5ad3c51b0a42f00bf0fad37c678fe54100722fcf92729fce5e792ba2dc3"} Nov 02 20:15:53.287866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:53.287695 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" event={"ID":"585fcd70-8a96-4a96-88b7-f088fad23b01","Type":"ContainerStarted","Data":"2acf0120825f6513623a56b0a7952d58e7917d57653d5d09efb0683231f7b424"} Nov 02 20:15:53.360638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:53.360604 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:53.367474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:53.367438 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:53.399133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:53.399081 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" podStartSLOduration=2.399065075 podStartE2EDuration="2.399065075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:53.395713874 +0000 UTC m=+1651.912486459" watchObservedRunningTime="2026-11-02 20:15:53.399065075 +0000 UTC m=+1651.915837661" Nov 02 20:15:54.295777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:54.295745 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:54.296889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:54.296865 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:15:56.233935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:56.233903 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:15:56.234465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:56.234006 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:15:59.725561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.725522 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm"] Nov 02 20:15:59.729979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.729954 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:15:59.782112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.782081 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm"] Nov 02 20:15:59.824718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.824673 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-40aecd\"" Nov 02 20:15:59.824991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.824968 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-40aecd-dockercfg-fj9c9\"" Nov 02 20:15:59.825561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.825516 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brstj\" (UniqueName: \"kubernetes.io/projected/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-kube-api-access-brstj\") pod \"payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:15:59.825815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.825787 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:15:59.910373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.909743 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp"] Nov 02 20:15:59.914948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.914920 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:15:59.926739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.926711 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:15:59.926879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.926846 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brstj\" (UniqueName: \"kubernetes.io/projected/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-kube-api-access-brstj\") pod \"payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:15:59.926933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.926918 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzw7k\" (UniqueName: \"kubernetes.io/projected/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-kube-api-access-bzw7k\") pod \"payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:15:59.927029 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.927011 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:15:59.929856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.929150 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:15:59.948271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.948239 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp"] Nov 02 20:15:59.982618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:15:59.982552 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brstj\" (UniqueName: \"kubernetes.io/projected/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-kube-api-access-brstj\") pod \"payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:16:00.030323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.030254 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw7k\" (UniqueName: \"kubernetes.io/projected/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-kube-api-access-bzw7k\") pod \"payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:16:00.030514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.030381 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:16:00.031090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.031068 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:16:00.045866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.045839 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:16:00.083429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.083398 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzw7k\" (UniqueName: \"kubernetes.io/projected/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-kube-api-access-bzw7k\") pod \"payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:16:00.233927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.233828 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:16:00.423752 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:16:00.423714 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6888dccd_94bb_4eb1_8fce_e4d6a79460ff.slice/crio-15af594e5116ccac6179d7097cefa396c0f4643137b80a855bcb203e07a5a951 WatchSource:0}: Error finding container 15af594e5116ccac6179d7097cefa396c0f4643137b80a855bcb203e07a5a951: Status 404 returned error can't find the container with id 15af594e5116ccac6179d7097cefa396c0f4643137b80a855bcb203e07a5a951 Nov 02 20:16:00.424427 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.424396 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm"] Nov 02 20:16:00.605189 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:00.605160 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp"] Nov 02 20:16:00.606034 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:16:00.606000 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0b9070_53dd_4f7f_bb2c_721b5d3f486c.slice/crio-a7c2a48e75c8d081c96d3bbc1f102b442677204f7720c3ea46f1a27f926ba11d WatchSource:0}: Error finding container a7c2a48e75c8d081c96d3bbc1f102b442677204f7720c3ea46f1a27f926ba11d: Status 404 returned error can't find the container with id a7c2a48e75c8d081c96d3bbc1f102b442677204f7720c3ea46f1a27f926ba11d Nov 02 20:16:01.355622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.355579 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" event={"ID":"6888dccd-94bb-4eb1-8fce-e4d6a79460ff","Type":"ContainerStarted","Data":"5d0fbe5d9ae39b2e1562c107e8d8f06f2d45d601c582c1d9b0cb180e5b7f6f54"} Nov 02 20:16:01.356133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.355639 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" event={"ID":"6888dccd-94bb-4eb1-8fce-e4d6a79460ff","Type":"ContainerStarted","Data":"15af594e5116ccac6179d7097cefa396c0f4643137b80a855bcb203e07a5a951"} Nov 02 20:16:01.356133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.355665 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:16:01.357618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.357592 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" event={"ID":"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c","Type":"ContainerStarted","Data":"110f672d7ed451d106f42f6ef36dd2c35e331f569c5e59f95741063685a92719"} Nov 02 20:16:01.357737 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.357638 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" event={"ID":"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c","Type":"ContainerStarted","Data":"a7c2a48e75c8d081c96d3bbc1f102b442677204f7720c3ea46f1a27f926ba11d"} Nov 02 20:16:01.357737 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.357690 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:16:01.461400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.461348 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" podStartSLOduration=2.46133034 podStartE2EDuration="2.46133034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:01.460253379 +0000 UTC m=+1659.977025965" watchObservedRunningTime="2026-11-02 20:16:01.46133034 +0000 UTC m=+1659.978102924" Nov 02 20:16:01.568746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:01.568689 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" podStartSLOduration=2.568669991 podStartE2EDuration="2.568669991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:01.567037614 +0000 UTC m=+1660.083810190" watchObservedRunningTime="2026-11-02 20:16:01.568669991 +0000 UTC m=+1660.085442575" Nov 02 20:16:12.367871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:12.367839 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:16:12.368343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:12.367919 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:16:18.536633 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:18.536596 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6"] Nov 02 20:16:18.635446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:18.635412 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g"] Nov 02 20:16:22.505662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:22.505629 5800 scope.go:122] "RemoveContainer" containerID="5a630d85374ffd2f0dc4253cd150601cde71e592cf2377820b1916bbd8936c28" Nov 02 20:16:22.519378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:22.519347 5800 scope.go:122] "RemoveContainer" containerID="1a9656d4fb4f27dc5f14b1e969ef2221a8b1cb8d78fdf7c6736bf014bb904ab5" Nov 02 20:16:22.542668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:22.542575 5800 scope.go:122] "RemoveContainer" containerID="9e49ba5bd6533ca3af124df28cae86e3306e7f4795be9ab956cca03ca20798a0" Nov 02 20:16:22.559680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:22.559657 5800 scope.go:122] "RemoveContainer" containerID="32d1532fb54a4d511e777ec3959f8d0e5cbef0e77f7c052e152e8985cd2730d4" Nov 02 20:16:22.573991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:22.573970 5800 scope.go:122] "RemoveContainer" containerID="5facfef97e52763e0f92628c6b927494d6bb9f468e148a9a660615ca47c6cc01" Nov 02 20:16:22.700206 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:22.700168 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:16:23.537034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.536955 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" podUID="78ce45cb-959d-4e11-bcb5-8012f7f51ee3" containerName="payload-processing" containerID="cri-o://952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82" gracePeriod=25 Nov 02 20:16:23.636324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.636272 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" podUID="5475d899-987e-4fd5-a697-5208740a47f4" containerName="payload-pre-processing" containerID="cri-o://e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f" gracePeriod=25 Nov 02 20:16:23.910596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.910566 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:16:23.975812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.975785 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-plugins-config-volume\") pod \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " Nov 02 20:16:23.976058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.976042 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-kube-api-access-hcgxt\" (UniqueName: \"kubernetes.io/projected/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-kube-api-access-hcgxt\") pod \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\" (UID: \"78ce45cb-959d-4e11-bcb5-8012f7f51ee3\") " Nov 02 20:16:23.976387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.976354 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-plugins-config-volume" pod "78ce45cb-959d-4e11-bcb5-8012f7f51ee3" (UID: "78ce45cb-959d-4e11-bcb5-8012f7f51ee3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:23.976577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.976555 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:23.978891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.978854 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-kube-api-access-hcgxt" pod "78ce45cb-959d-4e11-bcb5-8012f7f51ee3" (UID: "78ce45cb-959d-4e11-bcb5-8012f7f51ee3"). InnerVolumeSpecName "kube-api-access-hcgxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:23.997841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:23.997818 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:16:24.077122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.077091 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5475d899-987e-4fd5-a697-5208740a47f4-kube-api-access-nwq4p\" (UniqueName: \"kubernetes.io/projected/5475d899-987e-4fd5-a697-5208740a47f4-kube-api-access-nwq4p\") pod \"5475d899-987e-4fd5-a697-5208740a47f4\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " Nov 02 20:16:24.077352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.077249 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5475d899-987e-4fd5-a697-5208740a47f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5475d899-987e-4fd5-a697-5208740a47f4-plugins-config-volume\") pod \"5475d899-987e-4fd5-a697-5208740a47f4\" (UID: \"5475d899-987e-4fd5-a697-5208740a47f4\") " Nov 02 20:16:24.077651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.077627 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5475d899-987e-4fd5-a697-5208740a47f4-plugins-config-volume" pod "5475d899-987e-4fd5-a697-5208740a47f4" (UID: "5475d899-987e-4fd5-a697-5208740a47f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:24.077908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.077893 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgxt\" (UniqueName: \"kubernetes.io/projected/78ce45cb-959d-4e11-bcb5-8012f7f51ee3-kube-api-access-hcgxt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:24.077948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.077912 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5475d899-987e-4fd5-a697-5208740a47f4-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:24.079253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.079230 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5475d899-987e-4fd5-a697-5208740a47f4-kube-api-access-nwq4p" pod "5475d899-987e-4fd5-a697-5208740a47f4" (UID: "5475d899-987e-4fd5-a697-5208740a47f4"). InnerVolumeSpecName "kube-api-access-nwq4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:24.178957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.178874 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwq4p\" (UniqueName: \"kubernetes.io/projected/5475d899-987e-4fd5-a697-5208740a47f4-kube-api-access-nwq4p\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:24.554891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.554860 5800 generic.go:356] "Generic (PLEG): container finished" podID="5475d899-987e-4fd5-a697-5208740a47f4" containerID="e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f" exitCode=0 Nov 02 20:16:24.555376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.554907 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" Nov 02 20:16:24.555376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.554943 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" event={"ID":"5475d899-987e-4fd5-a697-5208740a47f4","Type":"ContainerDied","Data":"e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f"} Nov 02 20:16:24.555376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.554997 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g" event={"ID":"5475d899-987e-4fd5-a697-5208740a47f4","Type":"ContainerDied","Data":"474784081e9b8d4ddb9b049698ef4b10d21bcb3eb2a8f86196242715bda2e6fc"} Nov 02 20:16:24.555376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.555020 5800 scope.go:122] "RemoveContainer" containerID="e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f" Nov 02 20:16:24.556871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.556745 5800 generic.go:356] "Generic (PLEG): container finished" podID="78ce45cb-959d-4e11-bcb5-8012f7f51ee3" containerID="952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82" exitCode=0 Nov 02 20:16:24.556871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.556777 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" event={"ID":"78ce45cb-959d-4e11-bcb5-8012f7f51ee3","Type":"ContainerDied","Data":"952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82"} Nov 02 20:16:24.556871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.556807 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" Nov 02 20:16:24.556871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.556836 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6" event={"ID":"78ce45cb-959d-4e11-bcb5-8012f7f51ee3","Type":"ContainerDied","Data":"88c36ae40822b6f2484d269bfdc8a810af69fe1c0f43e242a3eb58e56d4ee4ab"} Nov 02 20:16:24.568711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.568686 5800 scope.go:122] "RemoveContainer" containerID="e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f" Nov 02 20:16:24.569028 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:24.569004 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f\": container with ID starting with e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f not found: ID does not exist" containerID="e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f" Nov 02 20:16:24.569117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.569034 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f"} err="failed to get container status \"e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f\": rpc error: code = NotFound desc = could not find container \"e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f\": container with ID starting with e9dbfefa024d39f5f69d3fab448d9289d88d0122b5640d96a0b0446db0b34b7f not found: ID does not exist" Nov 02 20:16:24.569117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.569051 5800 scope.go:122] "RemoveContainer" containerID="952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82" Nov 02 20:16:24.579177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.579159 5800 scope.go:122] "RemoveContainer" containerID="952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82" Nov 02 20:16:24.579462 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:24.579441 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82\": container with ID starting with 952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82 not found: ID does not exist" containerID="952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82" Nov 02 20:16:24.579560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.579469 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82"} err="failed to get container status \"952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82\": rpc error: code = NotFound desc = could not find container \"952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82\": container with ID starting with 952e9a17c72adcf97632eb1838d1bfc72543929ef2faeb924d19c7a5616cec82 not found: ID does not exist" Nov 02 20:16:24.709715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.709683 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6"] Nov 02 20:16:24.755989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.755956 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-671e7c-dc54bbbd5-zslx6"] Nov 02 20:16:24.813924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.813833 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g"] Nov 02 20:16:24.817035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:24.817012 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-671e7c-6d557846cf-bpn6g"] Nov 02 20:16:25.622651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:25.622613 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5475d899-987e-4fd5-a697-5208740a47f4" path="/var/lib/kubelet/pods/5475d899-987e-4fd5-a697-5208740a47f4/volumes" Nov 02 20:16:25.623079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:25.623064 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ce45cb-959d-4e11-bcb5-8012f7f51ee3" path="/var/lib/kubelet/pods/78ce45cb-959d-4e11-bcb5-8012f7f51ee3/volumes" Nov 02 20:16:39.184834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:39.184800 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf"] Nov 02 20:16:39.185443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:39.185012 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" containerID="cri-o://03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d" gracePeriod=30 Nov 02 20:16:40.646371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:40.646332 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c"] Nov 02 20:16:40.646812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:40.646594 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" containerID="cri-o://bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f" gracePeriod=30 Nov 02 20:16:41.438742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:41.438691 5800 patch_prober.go:29] interesting pod/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:16:41.438923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:41.438777 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:16:43.673349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.673318 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5"] Nov 02 20:16:43.673937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.673922 5800 cpu_manager.go:414] "containerMap: removing container" podUID="78ce45cb-959d-4e11-bcb5-8012f7f51ee3" containerName="payload-processing" Nov 02 20:16:43.673992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.673939 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ce45cb-959d-4e11-bcb5-8012f7f51ee3" containerName="payload-processing" Nov 02 20:16:43.673992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.673972 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5475d899-987e-4fd5-a697-5208740a47f4" containerName="payload-pre-processing" Nov 02 20:16:43.673992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.673977 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5475d899-987e-4fd5-a697-5208740a47f4" containerName="payload-pre-processing" Nov 02 20:16:43.674107 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.674094 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5475d899-987e-4fd5-a697-5208740a47f4" containerName="payload-pre-processing" Nov 02 20:16:43.674142 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.674108 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ce45cb-959d-4e11-bcb5-8012f7f51ee3" containerName="payload-processing" Nov 02 20:16:43.677531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.677515 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.730938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.730903 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5"] Nov 02 20:16:43.737544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.737485 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:16:43.738003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.737943 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:16:43.782942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.782906 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:16:43.782942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.782911 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:16:43.846512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.846480 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.846681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.846625 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.846681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.846676 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmcrl\" (UniqueName: \"kubernetes.io/projected/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-kube-api-access-xmcrl\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.947522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.947443 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.947522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.947497 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmcrl\" (UniqueName: \"kubernetes.io/projected/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-kube-api-access-xmcrl\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.947703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.947654 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.948262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.948234 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:43.949792 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:43.949773 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:44.000936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.000897 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmcrl\" (UniqueName: \"kubernetes.io/projected/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-kube-api-access-xmcrl\") pod \"maas-api-revoke-sub-72cf3860-8q7k5\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:44.291649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.291613 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:44.501443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.501416 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:16:44.657499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.657464 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-kube-api-access-bn5wf\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-kube-api-access-bn5wf\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.657706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.657524 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-credential-socket\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.657706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.657582 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-certs\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.657836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.657781 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-token\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-token\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.657903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.657831 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-credential-socket" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:44.657968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.657944 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-certs" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:44.658025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.657975 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-podinfo\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.658093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.658069 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-envoy\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.658327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.658162 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-socket\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.658327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.658208 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-data\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.658327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.658244 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istiod-ca-cert\") pod \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\" (UID: \"8782ae10-2e36-4da3-b1d3-cabb1ad325fa\") " Nov 02 20:16:44.659012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.658986 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.659108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.659017 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.659487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.659459 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istiod-ca-cert" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:44.659744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.659719 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-socket" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:44.659929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.659908 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-data" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:44.662515 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.662489 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-token" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:44.662622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.662555 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-podinfo" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:44.662622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.662608 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-envoy" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:44.662751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.662676 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-kube-api-access-bn5wf" pod "8782ae10-2e36-4da3-b1d3-cabb1ad325fa" (UID: "8782ae10-2e36-4da3-b1d3-cabb1ad325fa"). InnerVolumeSpecName "kube-api-access-bn5wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:44.682941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.682909 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5"] Nov 02 20:16:44.745825 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.745732 5800 generic.go:356] "Generic (PLEG): container finished" podID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerID="03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d" exitCode=0 Nov 02 20:16:44.745971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.745840 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" event={"ID":"8782ae10-2e36-4da3-b1d3-cabb1ad325fa","Type":"ContainerDied","Data":"03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d"} Nov 02 20:16:44.745971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.745896 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" event={"ID":"8782ae10-2e36-4da3-b1d3-cabb1ad325fa","Type":"ContainerDied","Data":"f11823ca05a81ccc0ec9dd4ec9af1af4284159cea838f0c6a0474d987869d4a8"} Nov 02 20:16:44.745971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.745919 5800 scope.go:122] "RemoveContainer" containerID="03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d" Nov 02 20:16:44.745971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.745848 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf" Nov 02 20:16:44.747463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.747437 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" event={"ID":"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10","Type":"ContainerStarted","Data":"00608d496e6fa6ec5e3b2d1b91afbd8acfa4ae2e95ccd00aa5a4dac2b2062822"} Nov 02 20:16:44.757789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.757769 5800 scope.go:122] "RemoveContainer" containerID="a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916" Nov 02 20:16:44.760528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.760509 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.760595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.760533 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.760595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.760548 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.760595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.760563 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.760595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.760578 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn5wf\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-kube-api-access-bn5wf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.760595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.760592 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.760754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.760605 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8782ae10-2e36-4da3-b1d3-cabb1ad325fa-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:44.769662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.769642 5800 scope.go:122] "RemoveContainer" containerID="03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d" Nov 02 20:16:44.769958 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:44.769937 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d\": container with ID starting with 03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d not found: ID does not exist" containerID="03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d" Nov 02 20:16:44.770031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.769970 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d"} err="failed to get container status \"03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d\": rpc error: code = NotFound desc = could not find container \"03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d\": container with ID starting with 03d3df95d3b418658f54a93d36abf49a1cf7136d889079bfc5d15e8a6510507d not found: ID does not exist" Nov 02 20:16:44.770031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.769995 5800 scope.go:122] "RemoveContainer" containerID="a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916" Nov 02 20:16:44.770256 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:44.770238 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916\": container with ID starting with a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916 not found: ID does not exist" containerID="a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916" Nov 02 20:16:44.770339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.770266 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916"} err="failed to get container status \"a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916\": rpc error: code = NotFound desc = could not find container \"a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916\": container with ID starting with a086abed17cd58539361dfb4a70920646eb1fc2dd43fe1e34e3accfbe84f7916 not found: ID does not exist" Nov 02 20:16:44.900953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.900918 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf"] Nov 02 20:16:44.904362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:44.904339 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a3021d-openshift-default-7f745f9f89-smrtf"] Nov 02 20:16:45.624560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:45.624524 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" path="/var/lib/kubelet/pods/8782ae10-2e36-4da3-b1d3-cabb1ad325fa/volumes" Nov 02 20:16:45.759456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:45.759414 5800 generic.go:356] "Generic (PLEG): container finished" podID="9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" containerID="eb96b29d54079a05650fb1c701b21a2a1bb7b3bc185db6e6dc4d7050654ee1b0" exitCode=0 Nov 02 20:16:45.759876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:45.759464 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" event={"ID":"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10","Type":"ContainerDied","Data":"eb96b29d54079a05650fb1c701b21a2a1bb7b3bc185db6e6dc4d7050654ee1b0"} Nov 02 20:16:45.938986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:45.938960 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:16:46.078341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078226 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-kube-api-access-mggw9\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-kube-api-access-mggw9\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.078341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078300 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-credential-socket\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.078553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078346 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-token\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.078553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078426 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-envoy\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.078726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078699 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-credential-socket" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:46.078917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078704 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-podinfo\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.079028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078933 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-certs\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.079028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.078987 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-data\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.079182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.079036 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-socket\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.079248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.079179 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693f9eea-24e2-446d-b5e9-cf8098787d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693f9eea-24e2-446d-b5e9-cf8098787d4a-istiod-ca-cert\") pod \"693f9eea-24e2-446d-b5e9-cf8098787d4a\" (UID: \"693f9eea-24e2-446d-b5e9-cf8098787d4a\") " Nov 02 20:16:46.079787 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.079762 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.080131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.080107 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693f9eea-24e2-446d-b5e9-cf8098787d4a-istiod-ca-cert" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:46.080560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.080536 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-data" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:46.080830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.080787 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-socket" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:46.080988 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.080969 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-certs" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:46.081677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.081636 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-envoy" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:46.081788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.081679 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-kube-api-access-mggw9" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "kube-api-access-mggw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:46.082199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.082176 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-podinfo" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:16:46.083385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.083363 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-token" pod "693f9eea-24e2-446d-b5e9-cf8098787d4a" (UID: "693f9eea-24e2-446d-b5e9-cf8098787d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:46.181265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181230 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.181265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181260 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.181265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181273 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.181548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181320 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.181548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181335 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.181548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181349 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/693f9eea-24e2-446d-b5e9-cf8098787d4a-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.181548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181360 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/693f9eea-24e2-446d-b5e9-cf8098787d4a-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.181548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.181370 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mggw9\" (UniqueName: \"kubernetes.io/projected/693f9eea-24e2-446d-b5e9-cf8098787d4a-kube-api-access-mggw9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:46.768874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.768837 5800 generic.go:356] "Generic (PLEG): container finished" podID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerID="bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f" exitCode=0 Nov 02 20:16:46.769327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.768885 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" Nov 02 20:16:46.769327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.768920 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" event={"ID":"693f9eea-24e2-446d-b5e9-cf8098787d4a","Type":"ContainerDied","Data":"bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f"} Nov 02 20:16:46.769327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.768980 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c" event={"ID":"693f9eea-24e2-446d-b5e9-cf8098787d4a","Type":"ContainerDied","Data":"99857c6e1590d562d3229f61403aba45ad77743d09eb5ea09c77b9093adcc1a9"} Nov 02 20:16:46.769327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.769010 5800 scope.go:122] "RemoveContainer" containerID="bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f" Nov 02 20:16:46.782393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.782373 5800 scope.go:122] "RemoveContainer" containerID="bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f" Nov 02 20:16:46.782724 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:46.782701 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f\": container with ID starting with bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f not found: ID does not exist" containerID="bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f" Nov 02 20:16:46.782815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.782731 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f"} err="failed to get container status \"bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f\": rpc error: code = NotFound desc = could not find container \"bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f\": container with ID starting with bbe04450584da85467198412a18910ba1d735c8e70192a1957e03fa6b6570a2f not found: ID does not exist" Nov 02 20:16:46.923457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.923420 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c"] Nov 02 20:16:46.938848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.938492 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:46.969090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:46.969060 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-671e7c-openshift-default-656dc9b847-4vj7c"] Nov 02 20:16:47.092642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.092532 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-xks-ca\") pod \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " Nov 02 20:16:47.092823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.092671 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-openshift-ca\") pod \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " Nov 02 20:16:47.093153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.093079 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-openshift-ca" pod "9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" (UID: "9c179dd9-0e57-475c-9fa9-a1e0ceff2e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:47.093563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.093536 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-kube-api-access-xmcrl\" (UniqueName: \"kubernetes.io/projected/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-kube-api-access-xmcrl\") pod \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\" (UID: \"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10\") " Nov 02 20:16:47.094178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.094155 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:47.095483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.095452 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-xks-ca" pod "9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" (UID: "9c179dd9-0e57-475c-9fa9-a1e0ceff2e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:47.095591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.095506 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-kube-api-access-xmcrl" pod "9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" (UID: "9c179dd9-0e57-475c-9fa9-a1e0ceff2e10"). InnerVolumeSpecName "kube-api-access-xmcrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:47.195423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.195384 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:47.195423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.195425 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmcrl\" (UniqueName: \"kubernetes.io/projected/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10-kube-api-access-xmcrl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:47.624602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.624571 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" path="/var/lib/kubelet/pods/693f9eea-24e2-446d-b5e9-cf8098787d4a/volumes" Nov 02 20:16:47.778253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.778224 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" Nov 02 20:16:47.778698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.778310 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5" event={"ID":"9c179dd9-0e57-475c-9fa9-a1e0ceff2e10","Type":"ContainerDied","Data":"00608d496e6fa6ec5e3b2d1b91afbd8acfa4ae2e95ccd00aa5a4dac2b2062822"} Nov 02 20:16:47.778698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:47.778367 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00608d496e6fa6ec5e3b2d1b91afbd8acfa4ae2e95ccd00aa5a4dac2b2062822" Nov 02 20:16:48.032128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:48.032091 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd"] Nov 02 20:16:48.135946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:48.135906 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk"] Nov 02 20:16:52.773632 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:52.773056 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:16:53.032981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.032884 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" podUID="5400ca29-d118-435b-b432-67931f51e264" containerName="payload-processing" containerID="cri-o://35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7" gracePeriod=25 Nov 02 20:16:53.136946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.136894 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" podUID="8de5a762-1a07-4969-9c1b-62c23463d813" containerName="payload-pre-processing" containerID="cri-o://7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0" gracePeriod=25 Nov 02 20:16:53.419720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.419675 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:16:53.482986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.482957 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:16:53.499631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.499605 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5400ca29-d118-435b-b432-67931f51e264-kube-api-access-rk9ds\" (UniqueName: \"kubernetes.io/projected/5400ca29-d118-435b-b432-67931f51e264-kube-api-access-rk9ds\") pod \"5400ca29-d118-435b-b432-67931f51e264\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " Nov 02 20:16:53.499769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.499665 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5400ca29-d118-435b-b432-67931f51e264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400ca29-d118-435b-b432-67931f51e264-plugins-config-volume\") pod \"5400ca29-d118-435b-b432-67931f51e264\" (UID: \"5400ca29-d118-435b-b432-67931f51e264\") " Nov 02 20:16:53.500062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.500035 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5400ca29-d118-435b-b432-67931f51e264-plugins-config-volume" pod "5400ca29-d118-435b-b432-67931f51e264" (UID: "5400ca29-d118-435b-b432-67931f51e264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:53.501811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.501790 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5400ca29-d118-435b-b432-67931f51e264-kube-api-access-rk9ds" pod "5400ca29-d118-435b-b432-67931f51e264" (UID: "5400ca29-d118-435b-b432-67931f51e264"). InnerVolumeSpecName "kube-api-access-rk9ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:53.601479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.601446 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de5a762-1a07-4969-9c1b-62c23463d813-kube-api-access-7wf6r\" (UniqueName: \"kubernetes.io/projected/8de5a762-1a07-4969-9c1b-62c23463d813-kube-api-access-7wf6r\") pod \"8de5a762-1a07-4969-9c1b-62c23463d813\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " Nov 02 20:16:53.601684 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.601594 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de5a762-1a07-4969-9c1b-62c23463d813-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5a762-1a07-4969-9c1b-62c23463d813-plugins-config-volume\") pod \"8de5a762-1a07-4969-9c1b-62c23463d813\" (UID: \"8de5a762-1a07-4969-9c1b-62c23463d813\") " Nov 02 20:16:53.602030 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.601999 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de5a762-1a07-4969-9c1b-62c23463d813-plugins-config-volume" pod "8de5a762-1a07-4969-9c1b-62c23463d813" (UID: "8de5a762-1a07-4969-9c1b-62c23463d813"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:53.602191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.602169 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk9ds\" (UniqueName: \"kubernetes.io/projected/5400ca29-d118-435b-b432-67931f51e264-kube-api-access-rk9ds\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.602262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.602193 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de5a762-1a07-4969-9c1b-62c23463d813-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.602262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.602203 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5400ca29-d118-435b-b432-67931f51e264-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.605037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.604417 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de5a762-1a07-4969-9c1b-62c23463d813-kube-api-access-7wf6r" pod "8de5a762-1a07-4969-9c1b-62c23463d813" (UID: "8de5a762-1a07-4969-9c1b-62c23463d813"). InnerVolumeSpecName "kube-api-access-7wf6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:53.703906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.703862 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wf6r\" (UniqueName: \"kubernetes.io/projected/8de5a762-1a07-4969-9c1b-62c23463d813-kube-api-access-7wf6r\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:53.824946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.824914 5800 generic.go:356] "Generic (PLEG): container finished" podID="8de5a762-1a07-4969-9c1b-62c23463d813" containerID="7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0" exitCode=0 Nov 02 20:16:53.825500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.824964 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" Nov 02 20:16:53.825500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.824994 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" event={"ID":"8de5a762-1a07-4969-9c1b-62c23463d813","Type":"ContainerDied","Data":"7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0"} Nov 02 20:16:53.825500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.825050 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk" event={"ID":"8de5a762-1a07-4969-9c1b-62c23463d813","Type":"ContainerDied","Data":"051e4a9f5e272d8613c85c32b726d3fa6f757db4cf27db5d670029d4f0b3b9ee"} Nov 02 20:16:53.825500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.825083 5800 scope.go:122] "RemoveContainer" containerID="7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0" Nov 02 20:16:53.827830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.827795 5800 generic.go:356] "Generic (PLEG): container finished" podID="5400ca29-d118-435b-b432-67931f51e264" containerID="35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7" exitCode=0 Nov 02 20:16:53.827977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.827963 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" Nov 02 20:16:53.828754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.828412 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" event={"ID":"5400ca29-d118-435b-b432-67931f51e264","Type":"ContainerDied","Data":"35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7"} Nov 02 20:16:53.828754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.828467 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd" event={"ID":"5400ca29-d118-435b-b432-67931f51e264","Type":"ContainerDied","Data":"e625201bc8e2b4847affe4a058e1ccc7e04a82f0db77e3f7a8cd242ea6270340"} Nov 02 20:16:53.838066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.838030 5800 scope.go:122] "RemoveContainer" containerID="7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0" Nov 02 20:16:53.838392 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:53.838363 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0\": container with ID starting with 7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0 not found: ID does not exist" containerID="7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0" Nov 02 20:16:53.838476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.838396 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0"} err="failed to get container status \"7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0\": rpc error: code = NotFound desc = could not find container \"7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0\": container with ID starting with 7b50245a5ae3ba917e05ed18ffa6f59b4352f1350a029f7caca3d14716ec39b0 not found: ID does not exist" Nov 02 20:16:53.838476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.838413 5800 scope.go:122] "RemoveContainer" containerID="35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7" Nov 02 20:16:53.848623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.848594 5800 scope.go:122] "RemoveContainer" containerID="35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7" Nov 02 20:16:53.848953 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:16:53.848933 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7\": container with ID starting with 35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7 not found: ID does not exist" containerID="35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7" Nov 02 20:16:53.849036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.848961 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7"} err="failed to get container status \"35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7\": rpc error: code = NotFound desc = could not find container \"35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7\": container with ID starting with 35b3fae276c7cba3e3d3729c2517fae1979d510a5fc7b5fda3ce2d5f524e11c7 not found: ID does not exist" Nov 02 20:16:53.979801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.979704 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk"] Nov 02 20:16:53.982773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:53.982748 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e0aa5c-5d989b7d67-5qcdk"] Nov 02 20:16:54.083011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:54.082978 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd"] Nov 02 20:16:54.090163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:54.090135 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e0aa5c-558955894c-zczgd"] Nov 02 20:16:55.626853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:55.626806 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5400ca29-d118-435b-b432-67931f51e264" path="/var/lib/kubelet/pods/5400ca29-d118-435b-b432-67931f51e264/volumes" Nov 02 20:16:55.627458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:16:55.627339 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de5a762-1a07-4969-9c1b-62c23463d813" path="/var/lib/kubelet/pods/8de5a762-1a07-4969-9c1b-62c23463d813/volumes" Nov 02 20:17:01.443530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.443419 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 02 20:17:01.626306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.615650 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 02 20:17:01.691306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.658494 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 20:17:01.691306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.662607 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:17:01.725364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.721536 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:17:01.730594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.730552 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:17:01.744451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.741608 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:17:01.744451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:01.742722 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:17:02.202725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.202689 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9"] Nov 02 20:17:02.203405 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203387 5800 cpu_manager.go:414] "containerMap: removing container" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203407 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203424 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5400ca29-d118-435b-b432-67931f51e264" containerName="payload-processing" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203431 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5400ca29-d118-435b-b432-67931f51e264" containerName="payload-processing" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203463 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" containerName="revoke-keys" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203471 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" containerName="revoke-keys" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203487 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8de5a762-1a07-4969-9c1b-62c23463d813" containerName="payload-pre-processing" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203494 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de5a762-1a07-4969-9c1b-62c23463d813" containerName="payload-pre-processing" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203524 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203532 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203561 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" Nov 02 20:17:02.203566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203568 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" Nov 02 20:17:02.204326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203720 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" Nov 02 20:17:02.204326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203740 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8782ae10-2e36-4da3-b1d3-cabb1ad325fa" containerName="istio-proxy" Nov 02 20:17:02.204326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203752 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="693f9eea-24e2-446d-b5e9-cf8098787d4a" containerName="istio-proxy" Nov 02 20:17:02.204326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203763 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5400ca29-d118-435b-b432-67931f51e264" containerName="payload-processing" Nov 02 20:17:02.204326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203775 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" containerName="revoke-keys" Nov 02 20:17:02.204326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.203786 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de5a762-1a07-4969-9c1b-62c23463d813" containerName="payload-pre-processing" Nov 02 20:17:02.209390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.209369 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.262990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.262951 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f83bc2-dockercfg-r89cq\"" Nov 02 20:17:02.277269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.277222 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsqq\" (UniqueName: \"kubernetes.io/projected/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-kube-api-access-8xsqq\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.277555 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.277532 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.284307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.281475 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9"] Nov 02 20:17:02.309872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.309737 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f83bc2\"" Nov 02 20:17:02.421308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.401190 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsqq\" (UniqueName: \"kubernetes.io/projected/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-kube-api-access-8xsqq\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.422466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.422428 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.423775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.403071 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8"] Nov 02 20:17:02.444082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.423582 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.444082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.437683 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:02.494371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.491594 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8"] Nov 02 20:17:02.499750 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.499719 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsqq\" (UniqueName: \"kubernetes.io/projected/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-kube-api-access-8xsqq\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.546368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.541952 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcpnt\" (UniqueName: \"kubernetes.io/projected/1e31de67-dae3-40e2-9b0e-de8b142eb24b-kube-api-access-hcpnt\") pod \"payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:02.546368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.542102 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e31de67-dae3-40e2-9b0e-de8b142eb24b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:02.546368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.542482 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:02.653476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.653307 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcpnt\" (UniqueName: \"kubernetes.io/projected/1e31de67-dae3-40e2-9b0e-de8b142eb24b-kube-api-access-hcpnt\") pod \"payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:02.654207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.653795 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e31de67-dae3-40e2-9b0e-de8b142eb24b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:02.656203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.656178 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e31de67-dae3-40e2-9b0e-de8b142eb24b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:02.754521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.747073 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcpnt\" (UniqueName: \"kubernetes.io/projected/1e31de67-dae3-40e2-9b0e-de8b142eb24b-kube-api-access-hcpnt\") pod \"payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:02.962919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:02.962877 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:03.022550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:03.021356 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9"] Nov 02 20:17:03.176327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:03.172912 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" event={"ID":"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1","Type":"ContainerStarted","Data":"92b11077871a085f3b5cb76e4a0c9e74ecab9906f71aa512520fb56bf627fab0"} Nov 02 20:17:03.364353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:03.363986 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8"] Nov 02 20:17:04.193526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:04.193487 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" event={"ID":"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1","Type":"ContainerStarted","Data":"49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf"} Nov 02 20:17:04.194035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:04.194019 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:04.195598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:04.195565 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" event={"ID":"1e31de67-dae3-40e2-9b0e-de8b142eb24b","Type":"ContainerStarted","Data":"0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c"} Nov 02 20:17:04.195706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:04.195636 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" event={"ID":"1e31de67-dae3-40e2-9b0e-de8b142eb24b","Type":"ContainerStarted","Data":"b732ce5c00ea146f420ccdd5b6dd59341e78a9ace8ac4916e743995b661250ea"} Nov 02 20:17:04.196148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:04.196127 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:04.305870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:04.305799 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" podStartSLOduration=2.305757951 podStartE2EDuration="2.305757951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:04.305432069 +0000 UTC m=+1722.822204656" watchObservedRunningTime="2026-11-02 20:17:04.305757951 +0000 UTC m=+1722.822530539" Nov 02 20:17:04.417308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:04.417073 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" podStartSLOduration=2.417052623 podStartE2EDuration="2.417052623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:04.416588262 +0000 UTC m=+1722.933360849" watchObservedRunningTime="2026-11-02 20:17:04.417052623 +0000 UTC m=+1722.933825209" Nov 02 20:17:07.173747 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.173709 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb"] Nov 02 20:17:07.178013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.177991 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.232647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.232612 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.232846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.232757 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68km\" (UniqueName: \"kubernetes.io/projected/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-kube-api-access-b68km\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.273598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.273555 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a999d7\"" Nov 02 20:17:07.274846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.274824 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a999d7-dockercfg-mkvcg\"" Nov 02 20:17:07.286376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.286345 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb"] Nov 02 20:17:07.334474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.334439 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68km\" (UniqueName: \"kubernetes.io/projected/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-kube-api-access-b68km\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.336967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.336496 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.337602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.337573 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.340890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.340866 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg"] Nov 02 20:17:07.363849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.363819 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.388414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.388386 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68km\" (UniqueName: \"kubernetes.io/projected/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-kube-api-access-b68km\") pod \"payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.407401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.407369 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg"] Nov 02 20:17:07.438491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.438388 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae247daf-d610-4e4d-ad75-044103a9291c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.438491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.438472 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfjt\" (UniqueName: \"kubernetes.io/projected/ae247daf-d610-4e4d-ad75-044103a9291c-kube-api-access-wkfjt\") pod \"payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.492197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.492166 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:07.540654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.540434 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae247daf-d610-4e4d-ad75-044103a9291c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.541316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.541257 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfjt\" (UniqueName: \"kubernetes.io/projected/ae247daf-d610-4e4d-ad75-044103a9291c-kube-api-access-wkfjt\") pod \"payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.541402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.541347 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae247daf-d610-4e4d-ad75-044103a9291c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.594714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.594670 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfjt\" (UniqueName: \"kubernetes.io/projected/ae247daf-d610-4e4d-ad75-044103a9291c-kube-api-access-wkfjt\") pod \"payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.692071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.691975 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:07.876129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:07.876101 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb"] Nov 02 20:17:07.876467 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:17:07.876445 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7bdb31_fb20_4254_be7e_fb70f0c70dfd.slice/crio-84d070dca3d95a9ba2e4f1e26065cba6a971ead3a5573c6714f65eba9615fb62 WatchSource:0}: Error finding container 84d070dca3d95a9ba2e4f1e26065cba6a971ead3a5573c6714f65eba9615fb62: Status 404 returned error can't find the container with id 84d070dca3d95a9ba2e4f1e26065cba6a971ead3a5573c6714f65eba9615fb62 Nov 02 20:17:08.073818 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:17:08.073775 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae247daf_d610_4e4d_ad75_044103a9291c.slice/crio-562f32e5371db05c0e438f1b9667b1ff2c6aefcc20a2d33bfce49ad634a64269 WatchSource:0}: Error finding container 562f32e5371db05c0e438f1b9667b1ff2c6aefcc20a2d33bfce49ad634a64269: Status 404 returned error can't find the container with id 562f32e5371db05c0e438f1b9667b1ff2c6aefcc20a2d33bfce49ad634a64269 Nov 02 20:17:08.074533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.074507 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg"] Nov 02 20:17:08.226216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.226130 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" event={"ID":"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd","Type":"ContainerStarted","Data":"25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99"} Nov 02 20:17:08.226216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.226185 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" event={"ID":"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd","Type":"ContainerStarted","Data":"84d070dca3d95a9ba2e4f1e26065cba6a971ead3a5573c6714f65eba9615fb62"} Nov 02 20:17:08.226762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.226309 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:08.227961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.227930 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" event={"ID":"ae247daf-d610-4e4d-ad75-044103a9291c","Type":"ContainerStarted","Data":"a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d"} Nov 02 20:17:08.228069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.227976 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" event={"ID":"ae247daf-d610-4e4d-ad75-044103a9291c","Type":"ContainerStarted","Data":"562f32e5371db05c0e438f1b9667b1ff2c6aefcc20a2d33bfce49ad634a64269"} Nov 02 20:17:08.228069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.228048 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:08.335741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.335685 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" podStartSLOduration=1.335670883 podStartE2EDuration="1.335670883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:08.332403068 +0000 UTC m=+1726.849175652" watchObservedRunningTime="2026-11-02 20:17:08.335670883 +0000 UTC m=+1726.852443468" Nov 02 20:17:08.443329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:08.441352 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" podStartSLOduration=1.44133296 podStartE2EDuration="1.44133296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:08.438312984 +0000 UTC m=+1726.955085569" watchObservedRunningTime="2026-11-02 20:17:08.44133296 +0000 UTC m=+1726.958105547" Nov 02 20:17:15.207199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:15.207165 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:17:15.207687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:15.207240 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:17:19.237924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:19.237892 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:17:19.238432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:19.237968 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:17:20.461048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.461010 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t"] Nov 02 20:17:20.465871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.465845 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:20.520620 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.520582 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t"] Nov 02 20:17:20.595316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.581026 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-296ae9-dockercfg-h6xwk\"" Nov 02 20:17:20.595316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.581099 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-296ae9\"" Nov 02 20:17:20.634194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.633691 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmrh\" (UniqueName: \"kubernetes.io/projected/9b5c97d6-a151-45e4-979d-7996b8782265-kube-api-access-mcmrh\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:20.634194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.633966 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c97d6-a151-45e4-979d-7996b8782265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:20.735714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.735617 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c97d6-a151-45e4-979d-7996b8782265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:20.735879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.735791 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmrh\" (UniqueName: \"kubernetes.io/projected/9b5c97d6-a151-45e4-979d-7996b8782265-kube-api-access-mcmrh\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:20.736508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.736483 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c97d6-a151-45e4-979d-7996b8782265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:20.791421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:20.791371 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmrh\" (UniqueName: \"kubernetes.io/projected/9b5c97d6-a151-45e4-979d-7996b8782265-kube-api-access-mcmrh\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:21.083187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:21.083139 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:21.460203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:21.460169 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t"] Nov 02 20:17:21.461520 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:17:21.461485 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5c97d6_a151_45e4_979d_7996b8782265.slice/crio-03467f94ddcf47e3daef26e3e821ed1d14d1ba0bef3e6be3d5bf1a75312f022a WatchSource:0}: Error finding container 03467f94ddcf47e3daef26e3e821ed1d14d1ba0bef3e6be3d5bf1a75312f022a: Status 404 returned error can't find the container with id 03467f94ddcf47e3daef26e3e821ed1d14d1ba0bef3e6be3d5bf1a75312f022a Nov 02 20:17:22.334772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:22.334730 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" event={"ID":"9b5c97d6-a151-45e4-979d-7996b8782265","Type":"ContainerStarted","Data":"9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38"} Nov 02 20:17:22.334966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:22.334798 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" event={"ID":"9b5c97d6-a151-45e4-979d-7996b8782265","Type":"ContainerStarted","Data":"03467f94ddcf47e3daef26e3e821ed1d14d1ba0bef3e6be3d5bf1a75312f022a"} Nov 02 20:17:22.334966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:22.334830 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:22.441627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:22.441567 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" podStartSLOduration=2.44155151 podStartE2EDuration="2.44155151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:22.43973803 +0000 UTC m=+1740.956510616" watchObservedRunningTime="2026-11-02 20:17:22.44155151 +0000 UTC m=+1740.958324094" Nov 02 20:17:22.701771 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:17:22.701737 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:17:30.359229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.359177 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg"] Nov 02 20:17:30.366330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.365761 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.394797 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.394761 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.394812 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee123445-e0bb-4395-a5ce-0829b4288d3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395011 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.394971 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztm9\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-kube-api-access-pztm9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.395016 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.395038 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.395110 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.395171 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.395235 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.395376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.395361 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497086 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497123 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497155 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497228 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497298 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497328 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee123445-e0bb-4395-a5ce-0829b4288d3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497377 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztm9\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-kube-api-access-pztm9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497401 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497419 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.497785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.497602 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.498184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.498154 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.498390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.498365 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.498868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.498842 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.499070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.499046 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee123445-e0bb-4395-a5ce-0829b4288d3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.500894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.500866 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.501002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.500952 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.557862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.557813 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg"] Nov 02 20:17:30.559123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.559093 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztm9\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-kube-api-access-pztm9\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.559824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.559791 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-dk9wg\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:30.683364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:30.683262 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:31.113511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.113478 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg"] Nov 02 20:17:31.114478 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:17:31.114453 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee123445_e0bb_4395_a5ce_0829b4288d3d.slice/crio-fd8bd451df43458d8e1d475e3dcfcb8b2efb7c01797b9d66aaa02ef9f5d7f701 WatchSource:0}: Error finding container fd8bd451df43458d8e1d475e3dcfcb8b2efb7c01797b9d66aaa02ef9f5d7f701: Status 404 returned error can't find the container with id fd8bd451df43458d8e1d475e3dcfcb8b2efb7c01797b9d66aaa02ef9f5d7f701 Nov 02 20:17:31.117008 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.116974 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:31.117112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.117040 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:31.117112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.117068 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:31.415973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.415863 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" event={"ID":"ee123445-e0bb-4395-a5ce-0829b4288d3d","Type":"ContainerStarted","Data":"13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e"} Nov 02 20:17:31.416479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.415975 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" event={"ID":"ee123445-e0bb-4395-a5ce-0829b4288d3d","Type":"ContainerStarted","Data":"fd8bd451df43458d8e1d475e3dcfcb8b2efb7c01797b9d66aaa02ef9f5d7f701"} Nov 02 20:17:31.531304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.528172 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podStartSLOduration=1.528156741 podStartE2EDuration="1.528156741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:31.523434154 +0000 UTC m=+1750.040206739" watchObservedRunningTime="2026-11-02 20:17:31.528156741 +0000 UTC m=+1750.044929398" Nov 02 20:17:31.684041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:31.683958 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:32.681184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.681144 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd"] Nov 02 20:17:32.686510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.686478 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 20:17:32.686665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.686533 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 20:17:32.686768 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.686716 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758610 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lr6\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-kube-api-access-n5lr6\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758671 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758703 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758753 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d550b163-002e-4427-932e-de780fe699c2-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758793 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758835 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d550b163-002e-4427-932e-de780fe699c2-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758894 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.758929 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.777302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.759020 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.792331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.789518 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd"] Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860662 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d550b163-002e-4427-932e-de780fe699c2-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860721 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860749 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d550b163-002e-4427-932e-de780fe699c2-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860787 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860805 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860846 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860966 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lr6\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-kube-api-access-n5lr6\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.860989 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.861005 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.861366 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.862306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.862093 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.863085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.862737 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.863085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.862774 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d550b163-002e-4427-932e-de780fe699c2-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.863085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.862993 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.865295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.865260 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d550b163-002e-4427-932e-de780fe699c2-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.865534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.865509 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.914777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.914741 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:32.914925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:32.914907 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lr6\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-kube-api-access-n5lr6\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:33.004754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.004723 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:33.362359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.358459 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:17:33.481937 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:17:33.481893 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd550b163_002e_4427_932e_de780fe699c2.slice/crio-026fb33cc49a8030eca8e75c5426663336d24fcb68cfcccef55dfa831036ed9a WatchSource:0}: Error finding container 026fb33cc49a8030eca8e75c5426663336d24fcb68cfcccef55dfa831036ed9a: Status 404 returned error can't find the container with id 026fb33cc49a8030eca8e75c5426663336d24fcb68cfcccef55dfa831036ed9a Nov 02 20:17:33.482577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.482392 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd"] Nov 02 20:17:33.484921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.484891 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:33.485026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.484963 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:33.485026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.485010 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:33.686034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.684749 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 20:17:33.686034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:33.684809 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 20:17:34.484257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:34.484209 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" event={"ID":"d550b163-002e-4427-932e-de780fe699c2","Type":"ContainerStarted","Data":"380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656"} Nov 02 20:17:34.484535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:34.484314 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" event={"ID":"d550b163-002e-4427-932e-de780fe699c2","Type":"ContainerStarted","Data":"026fb33cc49a8030eca8e75c5426663336d24fcb68cfcccef55dfa831036ed9a"} Nov 02 20:17:34.603774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:34.603690 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podStartSLOduration=2.603667409 podStartE2EDuration="2.603667409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:34.597374656 +0000 UTC m=+1753.114147243" watchObservedRunningTime="2026-11-02 20:17:34.603667409 +0000 UTC m=+1753.120439994" Nov 02 20:17:34.684128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:34.684079 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 20:17:34.684365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:34.684151 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 20:17:35.005673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:35.005637 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:35.007405 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:35.007374 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:17:35.007548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:35.007427 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:17:35.684323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:35.684265 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 20:17:35.684544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:35.684335 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 20:17:36.005757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:36.005717 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:17:36.006171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:36.005784 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:17:36.692829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:36.692741 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 20:17:36.693128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:36.693098 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 20:17:37.009332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.008884 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:17:37.009332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.008953 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:17:37.511916 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.511871 5800 generic.go:356] "Generic (PLEG): container finished" podID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerID="7d25e5ad3c51b0a42f00bf0fad37c678fe54100722fcf92729fce5e792ba2dc3" exitCode=0 Nov 02 20:17:37.512102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.511946 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" event={"ID":"585fcd70-8a96-4a96-88b7-f088fad23b01","Type":"ContainerDied","Data":"7d25e5ad3c51b0a42f00bf0fad37c678fe54100722fcf92729fce5e792ba2dc3"} Nov 02 20:17:37.512447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.512429 5800 scope.go:122] "RemoveContainer" containerID="7d25e5ad3c51b0a42f00bf0fad37c678fe54100722fcf92729fce5e792ba2dc3" Nov 02 20:17:37.513899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.513875 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:37.513983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.513942 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:37.514027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.513981 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:37.684200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.684165 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 20:17:37.684430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.684216 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 20:17:37.756316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.752782 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn"] Nov 02 20:17:37.789803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.781841 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.841614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.841575 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn"] Nov 02 20:17:37.909393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.909350 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.909595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.909500 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.909652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.909618 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.909693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.909667 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.909817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.909781 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2nf\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-kube-api-access-9t2nf\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.909912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.909892 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.909973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.909945 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.910062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.910001 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:37.910221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:37.910199 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.005782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.005742 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:17:38.005966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.005808 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:17:38.011544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.011519 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.011961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.011561 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.011961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.011594 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2nf\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-kube-api-access-9t2nf\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.011961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.011654 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.011961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.011695 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.011961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.011757 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012006 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012067 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012247 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012250 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012347 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012387 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012932 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.012962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.012951 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.014484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.014428 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.014484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.014452 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.065455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.065315 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.066301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.066262 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2nf\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-kube-api-access-9t2nf\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.179724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.179686 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:38.600613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.600572 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" event={"ID":"585fcd70-8a96-4a96-88b7-f088fad23b01","Type":"ContainerStarted","Data":"8dd226cffdf299c9eadbf67b5a2c956bc37ba9f45dd7dbda2d0f84d2afb6eda9"} Nov 02 20:17:38.686259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.686229 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 20:17:38.691258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.690376 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 20:17:38.691258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.689770 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:38.691258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.690498 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:38.691258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.690544 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:38.702362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:38.702301 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn"] Nov 02 20:17:39.005847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.005796 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:17:39.006049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.005860 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:17:39.296241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.296129 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:17:39.361831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.361798 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:17:39.362132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.362110 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 20:17:39.362315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.362275 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 20:17:39.705135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.705046 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" event={"ID":"9fc22d1f-0707-40dd-9527-6f88e8b89a7c","Type":"ContainerStarted","Data":"44752c2ee52358efece054d8b8f721d4f25efc3f42f4d2b94406a255ea97a174"} Nov 02 20:17:39.705554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.705530 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" event={"ID":"9fc22d1f-0707-40dd-9527-6f88e8b89a7c","Type":"ContainerStarted","Data":"6c382186dc015c9d1b29f403bae74aa87463aa9448faa352068e380e8ecb9fc7"} Nov 02 20:17:39.825811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:39.825748 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podStartSLOduration=2.825727541 podStartE2EDuration="2.825727541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:39.822340869 +0000 UTC m=+1758.339113454" watchObservedRunningTime="2026-11-02 20:17:39.825727541 +0000 UTC m=+1758.342500127" Nov 02 20:17:40.006914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.006878 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 02 20:17:40.007168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.007143 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 02 20:17:40.182777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.182730 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:17:40.183065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.183031 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:17:40.183863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.183844 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:40.362761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.362666 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 20:17:40.363334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.363302 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 20:17:40.799628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.798391 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:17:40.799628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:40.798474 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:17:41.020851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.019417 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:41.020851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.020663 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:41.023201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.023169 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:17:41.163966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.163868 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:41.165501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.165477 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:41.176708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.176677 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:17:41.183414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.183386 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:17:41.183640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.183621 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:17:41.393331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.393268 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 20:17:41.394002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:41.393970 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 20:17:42.187309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:42.181875 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:17:42.187309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:42.181937 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:17:42.372307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:42.369678 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 20:17:42.372307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:42.369741 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 20:17:43.180974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:43.180934 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:17:43.181575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:43.180990 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:17:43.369307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:43.362730 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 02 20:17:43.369307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:43.362798 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 02 20:17:44.180956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:44.180911 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:17:44.181136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:44.180981 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:17:44.366334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:44.366272 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:17:44.367649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:44.367626 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:17:45.181077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:45.181041 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 02 20:17:45.181263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:45.181096 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 02 20:17:46.185829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:46.185798 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:46.186382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:46.186297 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:46.187254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:46.187225 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:17:47.800378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.800338 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6"] Nov 02 20:17:47.805412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.805382 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.868629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.868573 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.868629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.868633 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.868902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.868718 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.868902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.868744 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.869055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.869035 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.869141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.869124 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.869202 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.869156 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.869264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.869230 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdtn\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-kube-api-access-7qdtn\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.869356 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.869330 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971117 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971160 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971196 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdtn\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-kube-api-access-7qdtn\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971367 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971473 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971504 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971621 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971646 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971678 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.971994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.971856 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.972542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.972423 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-certs\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.973162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.973130 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istiod-ca-cert\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.973315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.973239 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-data\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.973502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.973439 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-credential-socket\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.976183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.976158 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-podinfo\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.976805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.976768 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-envoy\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:47.993899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:47.993865 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6"] Nov 02 20:17:48.025868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.025833 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdtn\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-kube-api-access-7qdtn\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:48.037890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.037859 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-token\") pod \"e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:48.121433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.121337 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:48.552771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.552733 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6"] Nov 02 20:17:48.554361 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:17:48.554317 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9aeacf2_320e_44ad_93db_0e451fdd8a54.slice/crio-2f64456072ebe9934e97568d92b6cf5cb45ec448d972e0cdbe308bc9b21e1b19 WatchSource:0}: Error finding container 2f64456072ebe9934e97568d92b6cf5cb45ec448d972e0cdbe308bc9b21e1b19: Status 404 returned error can't find the container with id 2f64456072ebe9934e97568d92b6cf5cb45ec448d972e0cdbe308bc9b21e1b19 Nov 02 20:17:48.557652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.557613 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:48.557756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.557691 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:48.557756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.557723 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:17:48.844013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.843904 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" event={"ID":"d9aeacf2-320e-44ad-93db-0e451fdd8a54","Type":"ContainerStarted","Data":"3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606"} Nov 02 20:17:48.844013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.843974 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" event={"ID":"d9aeacf2-320e-44ad-93db-0e451fdd8a54","Type":"ContainerStarted","Data":"2f64456072ebe9934e97568d92b6cf5cb45ec448d972e0cdbe308bc9b21e1b19"} Nov 02 20:17:48.952233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:48.952170 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podStartSLOduration=1.9521511839999999 podStartE2EDuration="1.952151184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:17:48.951091754 +0000 UTC m=+1767.467864349" watchObservedRunningTime="2026-11-02 20:17:48.952151184 +0000 UTC m=+1767.468923769" Nov 02 20:17:49.122437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:49.122346 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:49.124042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:49.124013 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:17:49.124177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:49.124075 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 20:17:50.122772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:50.122733 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:17:50.123186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:50.122794 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 20:17:51.122111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:51.122068 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:17:51.122343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:51.122143 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 20:17:52.122552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:52.122513 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:17:52.122972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:52.122587 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 20:17:52.698009 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:17:52.697971 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:17:53.122485 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:53.122442 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:17:53.122708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:53.122506 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 20:17:54.122714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:54.122679 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:17:54.123182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:54.122753 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 20:17:55.121765 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:55.121727 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 02 20:17:55.121977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:55.121785 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 02 20:17:56.127169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:56.127136 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:56.127663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:56.127644 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:56.128523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:56.128501 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:17:56.502794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:17:56.502755 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27"] Nov 02 20:18:01.503837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.503794 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" podUID="3ead968e-a8c6-4d69-8856-1b932ee3e62c" containerName="payload-pre-processing" containerID="cri-o://dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed" gracePeriod=25 Nov 02 20:18:01.802159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.802126 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:18:01.949441 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.949408 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ead968e-a8c6-4d69-8856-1b932ee3e62c-kube-api-access-cfgml\" (UniqueName: \"kubernetes.io/projected/3ead968e-a8c6-4d69-8856-1b932ee3e62c-kube-api-access-cfgml\") pod \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " Nov 02 20:18:01.949635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.949483 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ead968e-a8c6-4d69-8856-1b932ee3e62c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead968e-a8c6-4d69-8856-1b932ee3e62c-plugins-config-volume\") pod \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\" (UID: \"3ead968e-a8c6-4d69-8856-1b932ee3e62c\") " Nov 02 20:18:01.949976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.949940 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ead968e-a8c6-4d69-8856-1b932ee3e62c-plugins-config-volume" pod "3ead968e-a8c6-4d69-8856-1b932ee3e62c" (UID: "3ead968e-a8c6-4d69-8856-1b932ee3e62c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:01.950234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.950214 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ead968e-a8c6-4d69-8856-1b932ee3e62c-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:01.951917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.951888 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ead968e-a8c6-4d69-8856-1b932ee3e62c-kube-api-access-cfgml" pod "3ead968e-a8c6-4d69-8856-1b932ee3e62c" (UID: "3ead968e-a8c6-4d69-8856-1b932ee3e62c"). InnerVolumeSpecName "kube-api-access-cfgml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:01.970628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.970600 5800 generic.go:356] "Generic (PLEG): container finished" podID="3ead968e-a8c6-4d69-8856-1b932ee3e62c" containerID="dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed" exitCode=0 Nov 02 20:18:01.970751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.970645 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" Nov 02 20:18:01.970751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.970668 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" event={"ID":"3ead968e-a8c6-4d69-8856-1b932ee3e62c","Type":"ContainerDied","Data":"dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed"} Nov 02 20:18:01.970751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.970721 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27" event={"ID":"3ead968e-a8c6-4d69-8856-1b932ee3e62c","Type":"ContainerDied","Data":"4a8f0a766a04bf9ad8a8326226ca725cc83228c537e83d614e8ef3073e443a67"} Nov 02 20:18:01.970751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.970742 5800 scope.go:122] "RemoveContainer" containerID="dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed" Nov 02 20:18:01.981853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.981830 5800 scope.go:122] "RemoveContainer" containerID="dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed" Nov 02 20:18:01.982136 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:01.982113 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed\": container with ID starting with dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed not found: ID does not exist" containerID="dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed" Nov 02 20:18:01.982187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:01.982145 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed"} err="failed to get container status \"dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed\": rpc error: code = NotFound desc = could not find container \"dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed\": container with ID starting with dcdb3aca527934d0aa9e2a50331a3bf0559f411054228dd4df9476d3684e82ed not found: ID does not exist" Nov 02 20:18:02.051906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:02.051819 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgml\" (UniqueName: \"kubernetes.io/projected/3ead968e-a8c6-4d69-8856-1b932ee3e62c-kube-api-access-cfgml\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:02.125805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:02.125771 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27"] Nov 02 20:18:02.129169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:02.129143 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-15203f-69cc6dd4f5-6pb27"] Nov 02 20:18:03.623554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:03.623518 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ead968e-a8c6-4d69-8856-1b932ee3e62c" path="/var/lib/kubelet/pods/3ead968e-a8c6-4d69-8856-1b932ee3e62c/volumes" Nov 02 20:18:12.752250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:12.752210 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs"] Nov 02 20:18:12.872119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:12.872079 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l"] Nov 02 20:18:17.753069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:17.753009 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" podUID="5ddd6607-38d2-4d08-ada6-3138fcdc84db" containerName="payload-processing" containerID="cri-o://3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a" gracePeriod=25 Nov 02 20:18:17.872641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:17.872599 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" podUID="b1d326b4-4235-4f45-a9f2-be2f5561d609" containerName="payload-pre-processing" containerID="cri-o://4a863d309587958a7f481b5e0e9a0bb0cf5bd2c01341e379f4170b0de248d87d" gracePeriod=25 Nov 02 20:18:18.102263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.102229 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:18:18.102763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.102732 5800 generic.go:356] "Generic (PLEG): container finished" podID="b1d326b4-4235-4f45-a9f2-be2f5561d609" containerID="4a863d309587958a7f481b5e0e9a0bb0cf5bd2c01341e379f4170b0de248d87d" exitCode=0 Nov 02 20:18:18.102923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.102807 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" event={"ID":"b1d326b4-4235-4f45-a9f2-be2f5561d609","Type":"ContainerDied","Data":"4a863d309587958a7f481b5e0e9a0bb0cf5bd2c01341e379f4170b0de248d87d"} Nov 02 20:18:18.104541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.104518 5800 generic.go:356] "Generic (PLEG): container finished" podID="5ddd6607-38d2-4d08-ada6-3138fcdc84db" containerID="3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a" exitCode=0 Nov 02 20:18:18.104667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.104571 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" Nov 02 20:18:18.104667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.104601 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" event={"ID":"5ddd6607-38d2-4d08-ada6-3138fcdc84db","Type":"ContainerDied","Data":"3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a"} Nov 02 20:18:18.104667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.104662 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs" event={"ID":"5ddd6607-38d2-4d08-ada6-3138fcdc84db","Type":"ContainerDied","Data":"bb096e4ce2c1bcd1f2d75a6ef1aeb5d2848212ad3a01795f24b5d94804239bf0"} Nov 02 20:18:18.104850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.104696 5800 scope.go:122] "RemoveContainer" containerID="3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a" Nov 02 20:18:18.115514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.115495 5800 scope.go:122] "RemoveContainer" containerID="3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a" Nov 02 20:18:18.115767 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:18.115743 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a\": container with ID starting with 3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a not found: ID does not exist" containerID="3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a" Nov 02 20:18:18.115863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.115777 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a"} err="failed to get container status \"3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a\": rpc error: code = NotFound desc = could not find container \"3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a\": container with ID starting with 3453ec8e9aca2d63ae7a696bca751a1c74ee7de80404787dfb7e0d91de8d260a not found: ID does not exist" Nov 02 20:18:18.131234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.131176 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ddd6607-38d2-4d08-ada6-3138fcdc84db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddd6607-38d2-4d08-ada6-3138fcdc84db-plugins-config-volume\") pod \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " Nov 02 20:18:18.131393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.131269 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ddd6607-38d2-4d08-ada6-3138fcdc84db-kube-api-access-wqqqc\" (UniqueName: \"kubernetes.io/projected/5ddd6607-38d2-4d08-ada6-3138fcdc84db-kube-api-access-wqqqc\") pod \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\" (UID: \"5ddd6607-38d2-4d08-ada6-3138fcdc84db\") " Nov 02 20:18:18.132682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.132655 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddd6607-38d2-4d08-ada6-3138fcdc84db-plugins-config-volume" pod "5ddd6607-38d2-4d08-ada6-3138fcdc84db" (UID: "5ddd6607-38d2-4d08-ada6-3138fcdc84db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:18.137588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.137559 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddd6607-38d2-4d08-ada6-3138fcdc84db-kube-api-access-wqqqc" pod "5ddd6607-38d2-4d08-ada6-3138fcdc84db" (UID: "5ddd6607-38d2-4d08-ada6-3138fcdc84db"). InnerVolumeSpecName "kube-api-access-wqqqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:18.162773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.162746 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:18:18.232319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.232273 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d326b4-4235-4f45-a9f2-be2f5561d609-kube-api-access-52wm5\" (UniqueName: \"kubernetes.io/projected/b1d326b4-4235-4f45-a9f2-be2f5561d609-kube-api-access-52wm5\") pod \"b1d326b4-4235-4f45-a9f2-be2f5561d609\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " Nov 02 20:18:18.232573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.232554 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d326b4-4235-4f45-a9f2-be2f5561d609-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d326b4-4235-4f45-a9f2-be2f5561d609-plugins-config-volume\") pod \"b1d326b4-4235-4f45-a9f2-be2f5561d609\" (UID: \"b1d326b4-4235-4f45-a9f2-be2f5561d609\") " Nov 02 20:18:18.233148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.233110 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d326b4-4235-4f45-a9f2-be2f5561d609-plugins-config-volume" pod "b1d326b4-4235-4f45-a9f2-be2f5561d609" (UID: "b1d326b4-4235-4f45-a9f2-be2f5561d609"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:18.233148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.233133 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ddd6607-38d2-4d08-ada6-3138fcdc84db-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:18.233309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.233158 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqqqc\" (UniqueName: \"kubernetes.io/projected/5ddd6607-38d2-4d08-ada6-3138fcdc84db-kube-api-access-wqqqc\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:18.234816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.234788 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d326b4-4235-4f45-a9f2-be2f5561d609-kube-api-access-52wm5" pod "b1d326b4-4235-4f45-a9f2-be2f5561d609" (UID: "b1d326b4-4235-4f45-a9f2-be2f5561d609"). InnerVolumeSpecName "kube-api-access-52wm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:18.335434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.335348 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d326b4-4235-4f45-a9f2-be2f5561d609-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:18.335434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.335382 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wm5\" (UniqueName: \"kubernetes.io/projected/b1d326b4-4235-4f45-a9f2-be2f5561d609-kube-api-access-52wm5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:18.519036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.518992 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs"] Nov 02 20:18:18.522420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:18.522393 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5e04421b-76bc586cdf-rm8gs"] Nov 02 20:18:19.117481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:19.117394 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" event={"ID":"b1d326b4-4235-4f45-a9f2-be2f5561d609","Type":"ContainerDied","Data":"cd9e6a58f0dbce57e02b0c62512fecb62bcf29806e06e99e7921728b1e7b85b6"} Nov 02 20:18:19.117481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:19.117417 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l" Nov 02 20:18:19.117481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:19.117455 5800 scope.go:122] "RemoveContainer" containerID="4a863d309587958a7f481b5e0e9a0bb0cf5bd2c01341e379f4170b0de248d87d" Nov 02 20:18:19.272673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:19.272638 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l"] Nov 02 20:18:19.320022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:19.319983 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5e04421b-555dbc6b4c-9rb6l"] Nov 02 20:18:19.635117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:19.635079 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ddd6607-38d2-4d08-ada6-3138fcdc84db" path="/var/lib/kubelet/pods/5ddd6607-38d2-4d08-ada6-3138fcdc84db/volumes" Nov 02 20:18:19.635550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:19.635534 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d326b4-4235-4f45-a9f2-be2f5561d609" path="/var/lib/kubelet/pods/b1d326b4-4235-4f45-a9f2-be2f5561d609/volumes" Nov 02 20:18:21.942443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:21.942412 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:18:21.953380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:21.953355 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:18:22.697021 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:22.696985 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:18:23.027430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.027393 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w"] Nov 02 20:18:23.028041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028027 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5ddd6607-38d2-4d08-ada6-3138fcdc84db" containerName="payload-processing" Nov 02 20:18:23.028085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028042 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ddd6607-38d2-4d08-ada6-3138fcdc84db" containerName="payload-processing" Nov 02 20:18:23.028085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028055 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3ead968e-a8c6-4d69-8856-1b932ee3e62c" containerName="payload-pre-processing" Nov 02 20:18:23.028085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028060 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ead968e-a8c6-4d69-8856-1b932ee3e62c" containerName="payload-pre-processing" Nov 02 20:18:23.028184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028090 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b1d326b4-4235-4f45-a9f2-be2f5561d609" containerName="payload-pre-processing" Nov 02 20:18:23.028184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028095 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d326b4-4235-4f45-a9f2-be2f5561d609" containerName="payload-pre-processing" Nov 02 20:18:23.028246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028200 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ddd6607-38d2-4d08-ada6-3138fcdc84db" containerName="payload-processing" Nov 02 20:18:23.028246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028211 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ead968e-a8c6-4d69-8856-1b932ee3e62c" containerName="payload-pre-processing" Nov 02 20:18:23.028246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.028221 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d326b4-4235-4f45-a9f2-be2f5561d609" containerName="payload-pre-processing" Nov 02 20:18:23.032968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.032948 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.123881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.123840 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.123976 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.124132 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124230 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.124186 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.124337 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6397d570-9ef1-47ca-a927-c397d174fe67-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.124477 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84zx\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-kube-api-access-h84zx\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.124611 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.124666 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6397d570-9ef1-47ca-a927-c397d174fe67-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.124754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.124723 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.223331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.223292 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w"] Nov 02 20:18:23.226233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226197 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226257 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226334 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6397d570-9ef1-47ca-a927-c397d174fe67-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226416 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84zx\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-kube-api-access-h84zx\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226520 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226549 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6397d570-9ef1-47ca-a927-c397d174fe67-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226587 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226631 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226692 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-certs\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.226716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.226704 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.227039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.227016 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-data\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.227598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.227572 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6397d570-9ef1-47ca-a927-c397d174fe67-istiod-ca-cert\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.228041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.228018 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.228226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.228199 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-credential-socket\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.230413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.230391 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-envoy\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.230914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.230887 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6397d570-9ef1-47ca-a927-c397d174fe67-istio-podinfo\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.279223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.279148 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-istio-token\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.279445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.279348 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84zx\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-kube-api-access-h84zx\") pod \"e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.346899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.346866 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:23.763109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.763078 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w"] Nov 02 20:18:23.767608 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:18:23.767579 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6397d570_9ef1_47ca_a927_c397d174fe67.slice/crio-604fa499f0310b10c51cc84f67f3559bd970ea4a79979d48a876f6b6ee4f4d07 WatchSource:0}: Error finding container 604fa499f0310b10c51cc84f67f3559bd970ea4a79979d48a876f6b6ee4f4d07: Status 404 returned error can't find the container with id 604fa499f0310b10c51cc84f67f3559bd970ea4a79979d48a876f6b6ee4f4d07 Nov 02 20:18:23.770333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.770262 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:23.770439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.770413 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:23.770505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:23.770465 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:18:24.164041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:24.163948 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" event={"ID":"6397d570-9ef1-47ca-a927-c397d174fe67","Type":"ContainerStarted","Data":"bc832d99da7b1a01ea4476292795053e47431339d37255c31baa775cc9884d51"} Nov 02 20:18:24.164041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:24.164010 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" event={"ID":"6397d570-9ef1-47ca-a927-c397d174fe67","Type":"ContainerStarted","Data":"604fa499f0310b10c51cc84f67f3559bd970ea4a79979d48a876f6b6ee4f4d07"} Nov 02 20:18:24.356492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:24.356451 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:24.358705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:24.358673 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:18:24.358884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:24.358726 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:18:24.367116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:24.367047 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podStartSLOduration=2.367033482 podStartE2EDuration="2.367033482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:18:24.364735887 +0000 UTC m=+1802.881508471" watchObservedRunningTime="2026-11-02 20:18:24.367033482 +0000 UTC m=+1802.883806124" Nov 02 20:18:25.350309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:25.347436 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:18:25.350309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:25.347511 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:18:26.350844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:26.350384 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:18:26.350844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:26.350458 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:18:27.350680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.350639 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:18:27.350979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.350945 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:18:27.484145 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.484088 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd"] Nov 02 20:18:27.484711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.484661 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" containerID="cri-o://380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656" gracePeriod=30 Nov 02 20:18:27.499951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.499922 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6"] Nov 02 20:18:27.501103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.501031 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" containerID="cri-o://3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606" gracePeriod=30 Nov 02 20:18:27.510368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.510343 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg"] Nov 02 20:18:27.510728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.510699 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" containerID="cri-o://e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca" gracePeriod=30 Nov 02 20:18:27.762484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.762445 5800 patch_prober.go:29] interesting pod/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:18:27.762784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:27.762753 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:18:28.350707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:28.350658 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:18:28.350896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:28.350725 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:18:29.350731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:29.350698 5800 patch_prober.go:29] interesting pod/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 20:18:29.351249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:29.350799 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 20:18:30.352475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:30.352390 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:30.352959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:30.352889 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:30.353712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:30.353679 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:18:32.850194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.850162 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:18:32.901403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.901367 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:18:32.926948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.926915 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-envoy\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.926960 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-certs\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927001 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-socket\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927029 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-data\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927078 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-istio-token\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-istio-token\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927109 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-podinfo\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927130 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-kube-api-access-7qdtn\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-kube-api-access-7qdtn\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927182 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istiod-ca-cert\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927225 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-envoy\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927258 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-credential-socket\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927342 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-token\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-token\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927425 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-data\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927491 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d550b163-002e-4427-932e-de780fe699c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d550b163-002e-4427-932e-de780fe699c2-istiod-ca-cert\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927550 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-certs\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927592 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d550b163-002e-4427-932e-de780fe699c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d550b163-002e-4427-932e-de780fe699c2-istio-podinfo\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927639 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-kube-api-access-n5lr6\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-kube-api-access-n5lr6\") pod \"d550b163-002e-4427-932e-de780fe699c2\" (UID: \"d550b163-002e-4427-932e-de780fe699c2\") " Nov 02 20:18:32.927816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927669 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-socket\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:32.927816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927772 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-certs" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.928079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.927723 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-socket" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.928079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928007 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-data" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.928199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928155 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-socket" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.928255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928241 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.928255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928253 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.928390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928296 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.928390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928308 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:32.928390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928311 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-certs" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.928921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928430 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istiod-ca-cert" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:32.928921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928516 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-data" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.928921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.928797 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d550b163-002e-4427-932e-de780fe699c2-istiod-ca-cert" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:32.932421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.932397 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-envoy" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.933125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.933102 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-credential-socket" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.943961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.943936 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-token" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:32.943961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.943943 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-podinfo" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:32.943961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.943952 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d550b163-002e-4427-932e-de780fe699c2-istio-podinfo" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:32.944176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.944017 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-istio-token" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:32.944176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.944136 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-envoy" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:32.944252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.944225 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-kube-api-access-n5lr6" pod "d550b163-002e-4427-932e-de780fe699c2" (UID: "d550b163-002e-4427-932e-de780fe699c2"). InnerVolumeSpecName "kube-api-access-n5lr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:32.945358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.945337 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-kube-api-access-7qdtn" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "kube-api-access-7qdtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:32.946094 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:32.946070 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:18:33.029333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029293 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-credential-socket\") pod \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\" (UID: \"d9aeacf2-320e-44ad-93db-0e451fdd8a54\") " Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029562 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-credential-socket" pod "d9aeacf2-320e-44ad-93db-0e451fdd8a54" (UID: "d9aeacf2-320e-44ad-93db-0e451fdd8a54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029855 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029877 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029891 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdtn\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-kube-api-access-7qdtn\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029907 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029921 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029975 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.029999 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.030014 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d550b163-002e-4427-932e-de780fe699c2-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.030030 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d550b163-002e-4427-932e-de780fe699c2-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.030046 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.030061 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.030075 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d550b163-002e-4427-932e-de780fe699c2-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.030096 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lr6\" (UniqueName: \"kubernetes.io/projected/d550b163-002e-4427-932e-de780fe699c2-kube-api-access-n5lr6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.030150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.030113 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9aeacf2-320e-44ad-93db-0e451fdd8a54-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.132085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132002 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-credential-socket\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132054 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-podinfo\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132089 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-certs\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132123 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-token\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-token\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132240 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-data\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132295 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-envoy\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132327 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-kube-api-access-hf9wl\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-kube-api-access-hf9wl\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132356 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-socket\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.132422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.132403 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istiod-ca-cert\") pod \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\" (UID: \"bb9856b5-90d2-4e34-84a7-43724bc9e1cd\") " Nov 02 20:18:33.133415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.133080 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-credential-socket" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.133415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.133304 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istiod-ca-cert" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:33.133415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.133277 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-certs" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.133415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.133371 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-socket" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.133751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.133547 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-data" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.138128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.138099 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-envoy" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:18:33.139650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.139614 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-podinfo" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:18:33.139650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.139644 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-kube-api-access-hf9wl" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "kube-api-access-hf9wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:33.140960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.140933 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-token" pod "bb9856b5-90d2-4e34-84a7-43724bc9e1cd" (UID: "bb9856b5-90d2-4e34-84a7-43724bc9e1cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:33.235081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235035 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.235458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235350 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.235458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235396 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9wl\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-kube-api-access-hf9wl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.235458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235415 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.235458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235429 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.235849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235785 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.235849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235810 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.235849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.235824 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.236165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.236109 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb9856b5-90d2-4e34-84a7-43724bc9e1cd-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:33.259700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.259663 5800 generic.go:356] "Generic (PLEG): container finished" podID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerID="e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca" exitCode=0 Nov 02 20:18:33.260111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.260083 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" event={"ID":"bb9856b5-90d2-4e34-84a7-43724bc9e1cd","Type":"ContainerDied","Data":"e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca"} Nov 02 20:18:33.260323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.260303 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" event={"ID":"bb9856b5-90d2-4e34-84a7-43724bc9e1cd","Type":"ContainerDied","Data":"3b9c6916b75b8d8aa4a4d8556bd169bd25a5369f245161d05f369000a660055f"} Nov 02 20:18:33.260736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.260707 5800 scope.go:122] "RemoveContainer" containerID="e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca" Nov 02 20:18:33.260929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.260913 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg" Nov 02 20:18:33.269709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.269675 5800 generic.go:356] "Generic (PLEG): container finished" podID="d550b163-002e-4427-932e-de780fe699c2" containerID="380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656" exitCode=0 Nov 02 20:18:33.269828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.269806 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" event={"ID":"d550b163-002e-4427-932e-de780fe699c2","Type":"ContainerDied","Data":"380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656"} Nov 02 20:18:33.269909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.269854 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" event={"ID":"d550b163-002e-4427-932e-de780fe699c2","Type":"ContainerDied","Data":"026fb33cc49a8030eca8e75c5426663336d24fcb68cfcccef55dfa831036ed9a"} Nov 02 20:18:33.269973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.269938 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd" Nov 02 20:18:33.273569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.272386 5800 generic.go:356] "Generic (PLEG): container finished" podID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerID="3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606" exitCode=0 Nov 02 20:18:33.273569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.272451 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" event={"ID":"d9aeacf2-320e-44ad-93db-0e451fdd8a54","Type":"ContainerDied","Data":"3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606"} Nov 02 20:18:33.273569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.272487 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" event={"ID":"d9aeacf2-320e-44ad-93db-0e451fdd8a54","Type":"ContainerDied","Data":"2f64456072ebe9934e97568d92b6cf5cb45ec448d972e0cdbe308bc9b21e1b19"} Nov 02 20:18:33.273569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.272563 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6" Nov 02 20:18:33.278471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.278449 5800 scope.go:122] "RemoveContainer" containerID="99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685" Nov 02 20:18:33.290626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.290605 5800 scope.go:122] "RemoveContainer" containerID="e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca" Nov 02 20:18:33.290920 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:33.290904 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca\": container with ID starting with e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca not found: ID does not exist" containerID="e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca" Nov 02 20:18:33.291001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.290929 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca"} err="failed to get container status \"e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca\": rpc error: code = NotFound desc = could not find container \"e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca\": container with ID starting with e54c909c331338060b0760dca4ece8f13b2671fba11bd104ce96662eb7c9dbca not found: ID does not exist" Nov 02 20:18:33.291001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.290956 5800 scope.go:122] "RemoveContainer" containerID="99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685" Nov 02 20:18:33.291226 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:33.291208 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685\": container with ID starting with 99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685 not found: ID does not exist" containerID="99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685" Nov 02 20:18:33.291323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.291234 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685"} err="failed to get container status \"99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685\": rpc error: code = NotFound desc = could not find container \"99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685\": container with ID starting with 99b8078481d3fe1fb343aa458f6b726556af21cdfaac883612b3ac2e519e4685 not found: ID does not exist" Nov 02 20:18:33.291323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.291253 5800 scope.go:122] "RemoveContainer" containerID="380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656" Nov 02 20:18:33.303840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.303814 5800 scope.go:122] "RemoveContainer" containerID="380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656" Nov 02 20:18:33.304117 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:33.304094 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656\": container with ID starting with 380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656 not found: ID does not exist" containerID="380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656" Nov 02 20:18:33.304206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.304122 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656"} err="failed to get container status \"380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656\": rpc error: code = NotFound desc = could not find container \"380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656\": container with ID starting with 380aa25dcb404e57dda06ac60f4b60858363397c19ce9075a72490023556a656 not found: ID does not exist" Nov 02 20:18:33.304206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.304137 5800 scope.go:122] "RemoveContainer" containerID="3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606" Nov 02 20:18:33.315056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.315030 5800 scope.go:122] "RemoveContainer" containerID="3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606" Nov 02 20:18:33.315355 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:33.315329 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606\": container with ID starting with 3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606 not found: ID does not exist" containerID="3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606" Nov 02 20:18:33.315438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.315364 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606"} err="failed to get container status \"3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606\": rpc error: code = NotFound desc = could not find container \"3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606\": container with ID starting with 3ebf730f82f0b735171500e8884585d5b67e095436a5cab9332dd40f5c6af606 not found: ID does not exist" Nov 02 20:18:33.420778 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.420688 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg"] Nov 02 20:18:33.423721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.423696 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-5cfjg"] Nov 02 20:18:33.530191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.530155 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6"] Nov 02 20:18:33.535423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.535396 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-qvps6"] Nov 02 20:18:33.632940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.632890 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" path="/var/lib/kubelet/pods/bb9856b5-90d2-4e34-84a7-43724bc9e1cd/volumes" Nov 02 20:18:33.635229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.635193 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" path="/var/lib/kubelet/pods/d9aeacf2-320e-44ad-93db-0e451fdd8a54/volumes" Nov 02 20:18:33.664595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.664338 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd"] Nov 02 20:18:33.678397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:33.672969 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e04421b-openshift-default-6df5dfdf89-7jsqd"] Nov 02 20:18:35.625359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:35.625323 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d550b163-002e-4427-932e-de780fe699c2" path="/var/lib/kubelet/pods/d550b163-002e-4427-932e-de780fe699c2/volumes" Nov 02 20:18:36.542476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.542425 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm"] Nov 02 20:18:36.543380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543360 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" Nov 02 20:18:36.543448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543384 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" Nov 02 20:18:36.543487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543450 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" Nov 02 20:18:36.543487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543458 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" Nov 02 20:18:36.543487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543482 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" Nov 02 20:18:36.543614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543489 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" Nov 02 20:18:36.543614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543510 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" Nov 02 20:18:36.543614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543520 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" Nov 02 20:18:36.543713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543662 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d550b163-002e-4427-932e-de780fe699c2" containerName="istio-proxy" Nov 02 20:18:36.543713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543676 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9aeacf2-320e-44ad-93db-0e451fdd8a54" containerName="istio-proxy" Nov 02 20:18:36.543713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543688 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" Nov 02 20:18:36.543713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.543699 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9856b5-90d2-4e34-84a7-43724bc9e1cd" containerName="istio-proxy" Nov 02 20:18:36.549581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.549558 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.597311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.597257 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm"] Nov 02 20:18:36.601532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.601499 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f66q\" (UniqueName: \"kubernetes.io/projected/463c4552-57cd-49d6-b1f2-b2f80621b910-kube-api-access-7f66q\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.601822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.601793 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.601979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.601885 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.639172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.638663 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:18:36.639172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.638691 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:18:36.644863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.644838 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:18:36.644863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.644838 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:18:36.702984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.702949 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f66q\" (UniqueName: \"kubernetes.io/projected/463c4552-57cd-49d6-b1f2-b2f80621b910-kube-api-access-7f66q\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.703185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.702999 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.703185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.703046 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.703758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.703735 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.705577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.705558 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.755849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.755814 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f66q\" (UniqueName: \"kubernetes.io/projected/463c4552-57cd-49d6-b1f2-b2f80621b910-kube-api-access-7f66q\") pod \"maas-api-revoke-sub-0cf06014-24wqm\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:36.864921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:36.864834 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:37.237917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:37.237880 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm"] Nov 02 20:18:37.239463 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:18:37.239430 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463c4552_57cd_49d6_b1f2_b2f80621b910.slice/crio-6da5ec025ec26b36e25dda739aeaf20d29bbbb1efae266aa1f445eaba47ec6ea WatchSource:0}: Error finding container 6da5ec025ec26b36e25dda739aeaf20d29bbbb1efae266aa1f445eaba47ec6ea: Status 404 returned error can't find the container with id 6da5ec025ec26b36e25dda739aeaf20d29bbbb1efae266aa1f445eaba47ec6ea Nov 02 20:18:37.305057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:37.305025 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" event={"ID":"463c4552-57cd-49d6-b1f2-b2f80621b910","Type":"ContainerStarted","Data":"6da5ec025ec26b36e25dda739aeaf20d29bbbb1efae266aa1f445eaba47ec6ea"} Nov 02 20:18:38.318153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:38.318112 5800 generic.go:356] "Generic (PLEG): container finished" podID="463c4552-57cd-49d6-b1f2-b2f80621b910" containerID="1c9cc9192719c8f78a1cf4f118a5e3619b9a1068fd79bff3bb4df4699705336c" exitCode=0 Nov 02 20:18:38.318685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:38.318213 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" event={"ID":"463c4552-57cd-49d6-b1f2-b2f80621b910","Type":"ContainerDied","Data":"1c9cc9192719c8f78a1cf4f118a5e3619b9a1068fd79bff3bb4df4699705336c"} Nov 02 20:18:39.485810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.485785 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:39.574519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.574472 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-openshift-ca\") pod \"463c4552-57cd-49d6-b1f2-b2f80621b910\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " Nov 02 20:18:39.574693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.574661 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-xks-ca\") pod \"463c4552-57cd-49d6-b1f2-b2f80621b910\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " Nov 02 20:18:39.574774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.574696 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463c4552-57cd-49d6-b1f2-b2f80621b910-kube-api-access-7f66q\" (UniqueName: \"kubernetes.io/projected/463c4552-57cd-49d6-b1f2-b2f80621b910-kube-api-access-7f66q\") pod \"463c4552-57cd-49d6-b1f2-b2f80621b910\" (UID: \"463c4552-57cd-49d6-b1f2-b2f80621b910\") " Nov 02 20:18:39.574849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.574825 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-openshift-ca" pod "463c4552-57cd-49d6-b1f2-b2f80621b910" (UID: "463c4552-57cd-49d6-b1f2-b2f80621b910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:39.575136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.575110 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:39.576861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.576806 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-xks-ca" pod "463c4552-57cd-49d6-b1f2-b2f80621b910" (UID: "463c4552-57cd-49d6-b1f2-b2f80621b910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:18:39.577039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.577025 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463c4552-57cd-49d6-b1f2-b2f80621b910-kube-api-access-7f66q" pod "463c4552-57cd-49d6-b1f2-b2f80621b910" (UID: "463c4552-57cd-49d6-b1f2-b2f80621b910"). InnerVolumeSpecName "kube-api-access-7f66q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:39.676233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.676204 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463c4552-57cd-49d6-b1f2-b2f80621b910-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:39.676233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:39.676231 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f66q\" (UniqueName: \"kubernetes.io/projected/463c4552-57cd-49d6-b1f2-b2f80621b910-kube-api-access-7f66q\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:40.333690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:40.333662 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" Nov 02 20:18:40.333865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:40.333701 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm" event={"ID":"463c4552-57cd-49d6-b1f2-b2f80621b910","Type":"ContainerDied","Data":"6da5ec025ec26b36e25dda739aeaf20d29bbbb1efae266aa1f445eaba47ec6ea"} Nov 02 20:18:40.333865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:40.333748 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da5ec025ec26b36e25dda739aeaf20d29bbbb1efae266aa1f445eaba47ec6ea" Nov 02 20:18:42.136391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:42.136345 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp"] Nov 02 20:18:42.248260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:42.248215 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm"] Nov 02 20:18:47.137311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.137200 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" podUID="9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" containerName="payload-processing" containerID="cri-o://110f672d7ed451d106f42f6ef36dd2c35e331f569c5e59f95741063685a92719" gracePeriod=25 Nov 02 20:18:47.249057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.249009 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" podUID="6888dccd-94bb-4eb1-8fce-e4d6a79460ff" containerName="payload-pre-processing" containerID="cri-o://5d0fbe5d9ae39b2e1562c107e8d8f06f2d45d601c582c1d9b0cb180e5b7f6f54" gracePeriod=25 Nov 02 20:18:47.389431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.389344 5800 generic.go:356] "Generic (PLEG): container finished" podID="9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" containerID="110f672d7ed451d106f42f6ef36dd2c35e331f569c5e59f95741063685a92719" exitCode=0 Nov 02 20:18:47.389588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.389567 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" event={"ID":"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c","Type":"ContainerDied","Data":"110f672d7ed451d106f42f6ef36dd2c35e331f569c5e59f95741063685a92719"} Nov 02 20:18:47.391502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.391472 5800 generic.go:356] "Generic (PLEG): container finished" podID="6888dccd-94bb-4eb1-8fce-e4d6a79460ff" containerID="5d0fbe5d9ae39b2e1562c107e8d8f06f2d45d601c582c1d9b0cb180e5b7f6f54" exitCode=0 Nov 02 20:18:47.391854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.391820 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" event={"ID":"6888dccd-94bb-4eb1-8fce-e4d6a79460ff","Type":"ContainerDied","Data":"5d0fbe5d9ae39b2e1562c107e8d8f06f2d45d601c582c1d9b0cb180e5b7f6f54"} Nov 02 20:18:47.445862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.445835 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:18:47.494816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.494783 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-plugins-config-volume\") pod \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " Nov 02 20:18:47.495090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.495070 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-kube-api-access-bzw7k\" (UniqueName: \"kubernetes.io/projected/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-kube-api-access-bzw7k\") pod \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\" (UID: \"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c\") " Nov 02 20:18:47.495184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.495155 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-plugins-config-volume" pod "9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" (UID: "9f0b9070-53dd-4f7f-bb2c-721b5d3f486c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:47.495914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.495892 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:47.506324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.497388 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-kube-api-access-bzw7k" pod "9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" (UID: "9f0b9070-53dd-4f7f-bb2c-721b5d3f486c"). InnerVolumeSpecName "kube-api-access-bzw7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:47.540673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.540644 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:18:47.596817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.596786 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-plugins-config-volume\") pod \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " Nov 02 20:18:47.596986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.596942 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-kube-api-access-brstj\" (UniqueName: \"kubernetes.io/projected/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-kube-api-access-brstj\") pod \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\" (UID: \"6888dccd-94bb-4eb1-8fce-e4d6a79460ff\") " Nov 02 20:18:47.597277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.597244 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-plugins-config-volume" pod "6888dccd-94bb-4eb1-8fce-e4d6a79460ff" (UID: "6888dccd-94bb-4eb1-8fce-e4d6a79460ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:18:47.597533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.597505 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:47.597533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.597534 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzw7k\" (UniqueName: \"kubernetes.io/projected/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c-kube-api-access-bzw7k\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:47.599078 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.599056 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-kube-api-access-brstj" pod "6888dccd-94bb-4eb1-8fce-e4d6a79460ff" (UID: "6888dccd-94bb-4eb1-8fce-e4d6a79460ff"). InnerVolumeSpecName "kube-api-access-brstj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:18:47.698810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:47.698770 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brstj\" (UniqueName: \"kubernetes.io/projected/6888dccd-94bb-4eb1-8fce-e4d6a79460ff-kube-api-access-brstj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:18:48.399877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.399844 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" Nov 02 20:18:48.400388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.399874 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp" event={"ID":"9f0b9070-53dd-4f7f-bb2c-721b5d3f486c","Type":"ContainerDied","Data":"a7c2a48e75c8d081c96d3bbc1f102b442677204f7720c3ea46f1a27f926ba11d"} Nov 02 20:18:48.400388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.399934 5800 scope.go:122] "RemoveContainer" containerID="110f672d7ed451d106f42f6ef36dd2c35e331f569c5e59f95741063685a92719" Nov 02 20:18:48.401628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.401608 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" Nov 02 20:18:48.401736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.401643 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm" event={"ID":"6888dccd-94bb-4eb1-8fce-e4d6a79460ff","Type":"ContainerDied","Data":"15af594e5116ccac6179d7097cefa396c0f4643137b80a855bcb203e07a5a951"} Nov 02 20:18:48.411296 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.411266 5800 scope.go:122] "RemoveContainer" containerID="5d0fbe5d9ae39b2e1562c107e8d8f06f2d45d601c582c1d9b0cb180e5b7f6f54" Nov 02 20:18:48.553771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.553735 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm"] Nov 02 20:18:48.600328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.600294 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-40aecd-68d8db5cc5-b8jcm"] Nov 02 20:18:48.656953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.656916 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp"] Nov 02 20:18:48.660532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:48.660508 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40aecd-b95dc9c6c-dfjqp"] Nov 02 20:18:49.623689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:49.623657 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6888dccd-94bb-4eb1-8fce-e4d6a79460ff" path="/var/lib/kubelet/pods/6888dccd-94bb-4eb1-8fce-e4d6a79460ff/volumes" Nov 02 20:18:49.624074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:49.624047 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" path="/var/lib/kubelet/pods/9f0b9070-53dd-4f7f-bb2c-721b5d3f486c/volumes" Nov 02 20:18:52.696716 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:18:52.696679 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:18:54.989885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:54.989852 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8"] Nov 02 20:18:55.091775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:55.091741 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9"] Nov 02 20:18:59.990255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:18:59.990207 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" podUID="1e31de67-dae3-40e2-9b0e-de8b142eb24b" containerName="payload-processing" containerID="cri-o://0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c" gracePeriod=25 Nov 02 20:19:00.092936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.092893 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" podUID="3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" containerName="payload-pre-processing" containerID="cri-o://49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf" gracePeriod=25 Nov 02 20:19:00.314805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.314751 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:19:00.390006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.389983 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:19:00.471581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.471545 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-plugins-config-volume\") pod \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " Nov 02 20:19:00.471884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.471863 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e31de67-dae3-40e2-9b0e-de8b142eb24b-kube-api-access-hcpnt\" (UniqueName: \"kubernetes.io/projected/1e31de67-dae3-40e2-9b0e-de8b142eb24b-kube-api-access-hcpnt\") pod \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " Nov 02 20:19:00.472073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.472056 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-kube-api-access-8xsqq\" (UniqueName: \"kubernetes.io/projected/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-kube-api-access-8xsqq\") pod \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\" (UID: \"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1\") " Nov 02 20:19:00.472217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.471971 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-plugins-config-volume" pod "3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" (UID: "3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:00.472413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.472396 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e31de67-dae3-40e2-9b0e-de8b142eb24b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e31de67-dae3-40e2-9b0e-de8b142eb24b-plugins-config-volume\") pod \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\" (UID: \"1e31de67-dae3-40e2-9b0e-de8b142eb24b\") " Nov 02 20:19:00.473108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.472801 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e31de67-dae3-40e2-9b0e-de8b142eb24b-plugins-config-volume" pod "1e31de67-dae3-40e2-9b0e-de8b142eb24b" (UID: "1e31de67-dae3-40e2-9b0e-de8b142eb24b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:00.473108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.473029 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:00.473108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.473051 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e31de67-dae3-40e2-9b0e-de8b142eb24b-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:00.475171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.475144 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-kube-api-access-8xsqq" pod "3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" (UID: "3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1"). InnerVolumeSpecName "kube-api-access-8xsqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:00.475327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.475181 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e31de67-dae3-40e2-9b0e-de8b142eb24b-kube-api-access-hcpnt" pod "1e31de67-dae3-40e2-9b0e-de8b142eb24b" (UID: "1e31de67-dae3-40e2-9b0e-de8b142eb24b"). InnerVolumeSpecName "kube-api-access-hcpnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:00.522129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.522098 5800 generic.go:356] "Generic (PLEG): container finished" podID="1e31de67-dae3-40e2-9b0e-de8b142eb24b" containerID="0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c" exitCode=0 Nov 02 20:19:00.522326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.522142 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" Nov 02 20:19:00.522326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.522189 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" event={"ID":"1e31de67-dae3-40e2-9b0e-de8b142eb24b","Type":"ContainerDied","Data":"0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c"} Nov 02 20:19:00.522326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.522252 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8" event={"ID":"1e31de67-dae3-40e2-9b0e-de8b142eb24b","Type":"ContainerDied","Data":"b732ce5c00ea146f420ccdd5b6dd59341e78a9ace8ac4916e743995b661250ea"} Nov 02 20:19:00.522326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.522304 5800 scope.go:122] "RemoveContainer" containerID="0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c" Nov 02 20:19:00.523950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.523924 5800 generic.go:356] "Generic (PLEG): container finished" podID="3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" containerID="49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf" exitCode=0 Nov 02 20:19:00.524076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.523974 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" Nov 02 20:19:00.524076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.523993 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" event={"ID":"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1","Type":"ContainerDied","Data":"49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf"} Nov 02 20:19:00.524076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.524051 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9" event={"ID":"3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1","Type":"ContainerDied","Data":"92b11077871a085f3b5cb76e4a0c9e74ecab9906f71aa512520fb56bf627fab0"} Nov 02 20:19:00.537760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.537736 5800 scope.go:122] "RemoveContainer" containerID="0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c" Nov 02 20:19:00.538093 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:00.538064 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c\": container with ID starting with 0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c not found: ID does not exist" containerID="0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c" Nov 02 20:19:00.538170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.538101 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c"} err="failed to get container status \"0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c\": rpc error: code = NotFound desc = could not find container \"0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c\": container with ID starting with 0c804c98c7f1ed5f23e4ebedadb526a37af742d7b63f89035471fc2e58bc5b7c not found: ID does not exist" Nov 02 20:19:00.538170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.538118 5800 scope.go:122] "RemoveContainer" containerID="49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf" Nov 02 20:19:00.551687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.551664 5800 scope.go:122] "RemoveContainer" containerID="49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf" Nov 02 20:19:00.551988 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:00.551957 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf\": container with ID starting with 49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf not found: ID does not exist" containerID="49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf" Nov 02 20:19:00.552074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.551998 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf"} err="failed to get container status \"49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf\": rpc error: code = NotFound desc = could not find container \"49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf\": container with ID starting with 49bbd61a973f39257d47a6eea2865cbe4f4ecb40131ffeaff1f7fa14452660cf not found: ID does not exist" Nov 02 20:19:00.574903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.574868 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcpnt\" (UniqueName: \"kubernetes.io/projected/1e31de67-dae3-40e2-9b0e-de8b142eb24b-kube-api-access-hcpnt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:00.574903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.574904 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xsqq\" (UniqueName: \"kubernetes.io/projected/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1-kube-api-access-8xsqq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:00.736301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.735443 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9"] Nov 02 20:19:00.739378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.739349 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-5d98ff7659-ngtl9"] Nov 02 20:19:00.837263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.837171 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8"] Nov 02 20:19:00.842096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:00.842060 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-5f6c4c8f57-sdst8"] Nov 02 20:19:01.625795 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:01.625757 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e31de67-dae3-40e2-9b0e-de8b142eb24b" path="/var/lib/kubelet/pods/1e31de67-dae3-40e2-9b0e-de8b142eb24b/volumes" Nov 02 20:19:01.626435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:01.626412 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" path="/var/lib/kubelet/pods/3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1/volumes" Nov 02 20:19:07.882329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.882268 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j"] Nov 02 20:19:07.883220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883196 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6888dccd-94bb-4eb1-8fce-e4d6a79460ff" containerName="payload-pre-processing" Nov 02 20:19:07.883277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883225 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6888dccd-94bb-4eb1-8fce-e4d6a79460ff" containerName="payload-pre-processing" Nov 02 20:19:07.883277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883274 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" containerName="payload-processing" Nov 02 20:19:07.883277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883299 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" containerName="payload-processing" Nov 02 20:19:07.883430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883349 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" containerName="payload-pre-processing" Nov 02 20:19:07.883430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883360 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" containerName="payload-pre-processing" Nov 02 20:19:07.883430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883372 5800 cpu_manager.go:414] "containerMap: removing container" podUID="463c4552-57cd-49d6-b1f2-b2f80621b910" containerName="revoke-keys" Nov 02 20:19:07.883430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883380 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463c4552-57cd-49d6-b1f2-b2f80621b910" containerName="revoke-keys" Nov 02 20:19:07.883430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883391 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1e31de67-dae3-40e2-9b0e-de8b142eb24b" containerName="payload-processing" Nov 02 20:19:07.883430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883399 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e31de67-dae3-40e2-9b0e-de8b142eb24b" containerName="payload-processing" Nov 02 20:19:07.883611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883547 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6888dccd-94bb-4eb1-8fce-e4d6a79460ff" containerName="payload-pre-processing" Nov 02 20:19:07.883611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883567 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0b9070-53dd-4f7f-bb2c-721b5d3f486c" containerName="payload-processing" Nov 02 20:19:07.883611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883585 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ceaed0b-9a55-4cd4-ac36-6a4ceb7361f1" containerName="payload-pre-processing" Nov 02 20:19:07.883611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883595 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="463c4552-57cd-49d6-b1f2-b2f80621b910" containerName="revoke-keys" Nov 02 20:19:07.883611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.883604 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e31de67-dae3-40e2-9b0e-de8b142eb24b" containerName="payload-processing" Nov 02 20:19:07.889744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.889716 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:07.944633 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:07.944581 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg"] Nov 02 20:19:08.012487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.004251 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-certs\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.016491 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-data\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.016580 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-socket\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.016609 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbcm\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-kube-api-access-znbcm\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.016646 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-token\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.016724 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-envoy\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.016755 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-credential-socket\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.016807 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istiod-ca-cert\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.017019 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-podinfo\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.020519 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d3184c-openshift-default-dockercfg-6z82x\"" Nov 02 20:19:08.040303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.027722 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j"] Nov 02 20:19:08.126419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.126376 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-podinfo\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.126621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.126446 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-certs\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.126621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.126512 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-data\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.126621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.126562 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-socket\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.126621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.126588 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbcm\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-kube-api-access-znbcm\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.126621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.126617 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-token\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.128599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.127634 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-data\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.128599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.127502 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-envoy\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.128599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.127944 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-credential-socket\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.128599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.127994 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istiod-ca-cert\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.128934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.128778 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istiod-ca-cert\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.128996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.128968 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-socket\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.129644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.127947 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-certs\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.129875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.129854 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-credential-socket\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.132892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.132832 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-podinfo\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.133527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.132939 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-envoy\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.134673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.134646 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb"] Nov 02 20:19:08.182712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.182677 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-token\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.182895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.182762 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbcm\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-kube-api-access-znbcm\") pod \"e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.228605 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.228574 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:08.393865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.393773 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc"] Nov 02 20:19:08.394089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.394048 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" containerID="cri-o://03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4" gracePeriod=30 Nov 02 20:19:08.677895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.677863 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j"] Nov 02 20:19:08.684012 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:19:08.683977 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60d3f55_ff2f_4876_8a0a_3a88340cefe1.slice/crio-0f371b0e2dacfef95b09c3df9fc93e8ddd9b5d9880deaba4b3e57994f20a6ba7 WatchSource:0}: Error finding container 0f371b0e2dacfef95b09c3df9fc93e8ddd9b5d9880deaba4b3e57994f20a6ba7: Status 404 returned error can't find the container with id 0f371b0e2dacfef95b09c3df9fc93e8ddd9b5d9880deaba4b3e57994f20a6ba7 Nov 02 20:19:08.686371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.686328 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:08.686463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.686423 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:08.686502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.686470 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:08.725611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:08.725568 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" event={"ID":"e60d3f55-ff2f-4876-8a0a-3a88340cefe1","Type":"ContainerStarted","Data":"0f371b0e2dacfef95b09c3df9fc93e8ddd9b5d9880deaba4b3e57994f20a6ba7"} Nov 02 20:19:09.735459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:09.735421 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" event={"ID":"e60d3f55-ff2f-4876-8a0a-3a88340cefe1","Type":"ContainerStarted","Data":"1d1f1d11b9c1a7b4dea4602eac0a8495fd7a077c9a8da1217d5608beb5d3e49c"} Nov 02 20:19:09.840504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:09.840460 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7"] Nov 02 20:19:09.840755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:09.840724 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" containerID="cri-o://173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2" gracePeriod=30 Nov 02 20:19:09.942006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:09.941952 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podStartSLOduration=2.9419380410000002 podStartE2EDuration="2.941938041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:09.891501886 +0000 UTC m=+1848.408274473" watchObservedRunningTime="2026-11-02 20:19:09.941938041 +0000 UTC m=+1848.458710626" Nov 02 20:19:10.229802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:10.229766 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:10.239009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:10.238972 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:10.765304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:10.764596 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:10.765781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:10.765491 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:19:12.334515 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:12.334477 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:19:12.334906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:12.334536 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:19:12.945693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:12.945649 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" podUID="ae247daf-d610-4e4d-ad75-044103a9291c" containerName="payload-processing" containerID="cri-o://a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d" gracePeriod=25 Nov 02 20:19:13.134986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.134942 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" podUID="fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" containerName="payload-pre-processing" containerID="cri-o://25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99" gracePeriod=25 Nov 02 20:19:13.178265 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:13.178221 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7bdb31_fb20_4254_be7e_fb70f0c70dfd.slice/crio-25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7bdb31_fb20_4254_be7e_fb70f0c70dfd.slice/crio-conmon-25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:19:13.274652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.274602 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:19:13.340428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.340387 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae247daf-d610-4e4d-ad75-044103a9291c-kube-api-access-wkfjt\" (UniqueName: \"kubernetes.io/projected/ae247daf-d610-4e4d-ad75-044103a9291c-kube-api-access-wkfjt\") pod \"ae247daf-d610-4e4d-ad75-044103a9291c\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " Nov 02 20:19:13.340930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.340459 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae247daf-d610-4e4d-ad75-044103a9291c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae247daf-d610-4e4d-ad75-044103a9291c-plugins-config-volume\") pod \"ae247daf-d610-4e4d-ad75-044103a9291c\" (UID: \"ae247daf-d610-4e4d-ad75-044103a9291c\") " Nov 02 20:19:13.341707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.341661 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae247daf-d610-4e4d-ad75-044103a9291c-plugins-config-volume" pod "ae247daf-d610-4e4d-ad75-044103a9291c" (UID: "ae247daf-d610-4e4d-ad75-044103a9291c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:13.343766 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.343735 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae247daf-d610-4e4d-ad75-044103a9291c-kube-api-access-wkfjt" pod "ae247daf-d610-4e4d-ad75-044103a9291c" (UID: "ae247daf-d610-4e4d-ad75-044103a9291c"). InnerVolumeSpecName "kube-api-access-wkfjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:13.391879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.391841 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:19:13.441704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.441675 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-plugins-config-volume\") pod \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " Nov 02 20:19:13.441870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.441760 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-kube-api-access-b68km\" (UniqueName: \"kubernetes.io/projected/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-kube-api-access-b68km\") pod \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\" (UID: \"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd\") " Nov 02 20:19:13.442176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.442143 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-plugins-config-volume" pod "fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" (UID: "fb7bdb31-fb20-4254-be7e-fb70f0c70dfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:13.442413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.442387 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfjt\" (UniqueName: \"kubernetes.io/projected/ae247daf-d610-4e4d-ad75-044103a9291c-kube-api-access-wkfjt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.442541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.442417 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae247daf-d610-4e4d-ad75-044103a9291c-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.442541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.442432 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.444151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.444121 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-kube-api-access-b68km" pod "fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" (UID: "fb7bdb31-fb20-4254-be7e-fb70f0c70dfd"). InnerVolumeSpecName "kube-api-access-b68km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:13.544604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.544571 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b68km\" (UniqueName: \"kubernetes.io/projected/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd-kube-api-access-b68km\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.690658 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.690630 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:19:13.746922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.746893 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-kube-api-access-ttnjf\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-kube-api-access-ttnjf\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.746988 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-credential-socket\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747012 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-certs\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747031 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-data\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747077 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-envoy\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747221 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-podinfo\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747294 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-token\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-token\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747345 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-socket\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747425 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istiod-ca-cert\") pod \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\" (UID: \"b8398ad9-e519-4c49-a40e-1efedaf04d5d\") " Nov 02 20:19:13.747728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747338 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-certs" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:13.747728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747355 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-credential-socket" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:13.747728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747547 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-data" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:13.747919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747702 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-socket" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:13.748004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.747982 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istiod-ca-cert" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:13.748371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.748168 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.748371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.748191 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.748371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.748211 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.748371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.748228 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.748371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.748242 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.749780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.749750 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-kube-api-access-ttnjf" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "kube-api-access-ttnjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:13.749945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.749923 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-envoy" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:13.750728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.750705 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-podinfo" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:13.750943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.750929 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-token" pod "b8398ad9-e519-4c49-a40e-1efedaf04d5d" (UID: "b8398ad9-e519-4c49-a40e-1efedaf04d5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:13.777408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.777327 5800 generic.go:356] "Generic (PLEG): container finished" podID="fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" containerID="25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99" exitCode=0 Nov 02 20:19:13.777533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.777429 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" Nov 02 20:19:13.777533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.777436 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" event={"ID":"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd","Type":"ContainerDied","Data":"25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99"} Nov 02 20:19:13.777533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.777491 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb" event={"ID":"fb7bdb31-fb20-4254-be7e-fb70f0c70dfd","Type":"ContainerDied","Data":"84d070dca3d95a9ba2e4f1e26065cba6a971ead3a5573c6714f65eba9615fb62"} Nov 02 20:19:13.777533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.777519 5800 scope.go:122] "RemoveContainer" containerID="25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99" Nov 02 20:19:13.779136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.779115 5800 generic.go:356] "Generic (PLEG): container finished" podID="ae247daf-d610-4e4d-ad75-044103a9291c" containerID="a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d" exitCode=0 Nov 02 20:19:13.779255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.779188 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" Nov 02 20:19:13.779255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.779211 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" event={"ID":"ae247daf-d610-4e4d-ad75-044103a9291c","Type":"ContainerDied","Data":"a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d"} Nov 02 20:19:13.779402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.779266 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg" event={"ID":"ae247daf-d610-4e4d-ad75-044103a9291c","Type":"ContainerDied","Data":"562f32e5371db05c0e438f1b9667b1ff2c6aefcc20a2d33bfce49ad634a64269"} Nov 02 20:19:13.781742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.781717 5800 generic.go:356] "Generic (PLEG): container finished" podID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerID="03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4" exitCode=0 Nov 02 20:19:13.781834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.781773 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" Nov 02 20:19:13.781933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.781914 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" event={"ID":"b8398ad9-e519-4c49-a40e-1efedaf04d5d","Type":"ContainerDied","Data":"03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4"} Nov 02 20:19:13.782138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.781958 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc" event={"ID":"b8398ad9-e519-4c49-a40e-1efedaf04d5d","Type":"ContainerDied","Data":"d8a1e9d73cce84b77796c360250c8002b143745bfab4e9b93e31c947c1a77adf"} Nov 02 20:19:13.788573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.788556 5800 scope.go:122] "RemoveContainer" containerID="25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99" Nov 02 20:19:13.788871 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:13.788845 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99\": container with ID starting with 25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99 not found: ID does not exist" containerID="25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99" Nov 02 20:19:13.788960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.788876 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99"} err="failed to get container status \"25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99\": rpc error: code = NotFound desc = could not find container \"25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99\": container with ID starting with 25ba8006c99368a361d6b6eb54d11ef74ee588bb1d11e9dc84a0ffcece426c99 not found: ID does not exist" Nov 02 20:19:13.788960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.788892 5800 scope.go:122] "RemoveContainer" containerID="a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d" Nov 02 20:19:13.798510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.798492 5800 scope.go:122] "RemoveContainer" containerID="a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d" Nov 02 20:19:13.798816 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:13.798797 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d\": container with ID starting with a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d not found: ID does not exist" containerID="a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d" Nov 02 20:19:13.798881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.798822 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d"} err="failed to get container status \"a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d\": rpc error: code = NotFound desc = could not find container \"a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d\": container with ID starting with a755e669a693be2b1fde76a716e5612dd7236f3eeacd871f70ca5ff83cb4491d not found: ID does not exist" Nov 02 20:19:13.798881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.798837 5800 scope.go:122] "RemoveContainer" containerID="03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4" Nov 02 20:19:13.815526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.815504 5800 scope.go:122] "RemoveContainer" containerID="815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810" Nov 02 20:19:13.828321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.828268 5800 scope.go:122] "RemoveContainer" containerID="03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4" Nov 02 20:19:13.828816 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:13.828789 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4\": container with ID starting with 03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4 not found: ID does not exist" containerID="03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4" Nov 02 20:19:13.828974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.828949 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4"} err="failed to get container status \"03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4\": rpc error: code = NotFound desc = could not find container \"03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4\": container with ID starting with 03e35d3d3b96acdbe5be073eb25e24aaa86286309adc9bc5136a7144985732c4 not found: ID does not exist" Nov 02 20:19:13.829080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.829067 5800 scope.go:122] "RemoveContainer" containerID="815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810" Nov 02 20:19:13.829683 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:13.829535 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810\": container with ID starting with 815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810 not found: ID does not exist" containerID="815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810" Nov 02 20:19:13.829683 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.829566 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810"} err="failed to get container status \"815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810\": rpc error: code = NotFound desc = could not find container \"815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810\": container with ID starting with 815dd59be3d6eada83a66857c69a458e179489eac0fac3685587b2745fc6d810 not found: ID does not exist" Nov 02 20:19:13.850093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.850050 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.850093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.850095 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.850461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.850111 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttnjf\" (UniqueName: \"kubernetes.io/projected/b8398ad9-e519-4c49-a40e-1efedaf04d5d-kube-api-access-ttnjf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.850461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.850127 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398ad9-e519-4c49-a40e-1efedaf04d5d-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:13.907087 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.907048 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb"] Nov 02 20:19:13.910126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.910098 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a999d7-7f794db867-c49sb"] Nov 02 20:19:13.943122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.943085 5800 patch_prober.go:29] interesting pod/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:19:13.943273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:13.943154 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:19:14.022459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:14.016968 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc"] Nov 02 20:19:14.023397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:14.023137 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-4hjfc"] Nov 02 20:19:14.131261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:14.131175 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg"] Nov 02 20:19:14.144762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:14.144729 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-7d5898579-5drfg"] Nov 02 20:19:15.125874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.125813 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:19:15.268246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268206 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-socket\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268335 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-envoy\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268404 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-credential-socket\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268433 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-certs\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268503 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69834227-930d-4f3b-af29-8e3d93c76e81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69834227-930d-4f3b-af29-8e3d93c76e81-istio-podinfo\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268534 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69834227-930d-4f3b-af29-8e3d93c76e81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69834227-930d-4f3b-af29-8e3d93c76e81-istiod-ca-cert\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268579 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-kube-api-access-752f2\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-kube-api-access-752f2\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268595 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-socket" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.268662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268638 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-data\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268685 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-istio-token\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-istio-token\") pod \"69834227-930d-4f3b-af29-8e3d93c76e81\" (UID: \"69834227-930d-4f3b-af29-8e3d93c76e81\") " Nov 02 20:19:15.268923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268772 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-credential-socket" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.268923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268803 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-certs" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.269141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.268937 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69834227-930d-4f3b-af29-8e3d93c76e81-istiod-ca-cert" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:15.269200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.269188 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.269257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.269193 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-data" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.269257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.269204 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.269371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.269258 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69834227-930d-4f3b-af29-8e3d93c76e81-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.269371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.269277 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.270758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.270739 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69834227-930d-4f3b-af29-8e3d93c76e81-istio-podinfo" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:15.270964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.270947 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-istio-token" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:15.271021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.270978 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-envoy" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:15.271122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.271104 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-kube-api-access-752f2" pod "69834227-930d-4f3b-af29-8e3d93c76e81" (UID: "69834227-930d-4f3b-af29-8e3d93c76e81"). InnerVolumeSpecName "kube-api-access-752f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:15.370445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.370409 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.370445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.370442 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69834227-930d-4f3b-af29-8e3d93c76e81-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.370660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.370456 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752f2\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-kube-api-access-752f2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.370660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.370471 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69834227-930d-4f3b-af29-8e3d93c76e81-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.370660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.370486 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69834227-930d-4f3b-af29-8e3d93c76e81-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:15.622642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.622605 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae247daf-d610-4e4d-ad75-044103a9291c" path="/var/lib/kubelet/pods/ae247daf-d610-4e4d-ad75-044103a9291c/volumes" Nov 02 20:19:15.623159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.623143 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" path="/var/lib/kubelet/pods/b8398ad9-e519-4c49-a40e-1efedaf04d5d/volumes" Nov 02 20:19:15.623583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.623567 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" path="/var/lib/kubelet/pods/fb7bdb31-fb20-4254-be7e-fb70f0c70dfd/volumes" Nov 02 20:19:15.825051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.824457 5800 generic.go:356] "Generic (PLEG): container finished" podID="69834227-930d-4f3b-af29-8e3d93c76e81" containerID="173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2" exitCode=0 Nov 02 20:19:15.825051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.824642 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" event={"ID":"69834227-930d-4f3b-af29-8e3d93c76e81","Type":"ContainerDied","Data":"173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2"} Nov 02 20:19:15.825051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.824693 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" event={"ID":"69834227-930d-4f3b-af29-8e3d93c76e81","Type":"ContainerDied","Data":"dfe4330e1b463bafe7def424a92fc7c5f3b525755963d4e922ad20865d51a672"} Nov 02 20:19:15.825051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.824723 5800 scope.go:122] "RemoveContainer" containerID="173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2" Nov 02 20:19:15.825051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.824848 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7" Nov 02 20:19:15.839159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.839133 5800 scope.go:122] "RemoveContainer" containerID="173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2" Nov 02 20:19:15.839542 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:15.839520 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2\": container with ID starting with 173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2 not found: ID does not exist" containerID="173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2" Nov 02 20:19:15.839627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:15.839551 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2"} err="failed to get container status \"173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2\": rpc error: code = NotFound desc = could not find container \"173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2\": container with ID starting with 173a19c80c874c004e11ffe28fac791b11bd03acd7c81514a0a2d213dd498ce2 not found: ID does not exist" Nov 02 20:19:16.026500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:16.026468 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7"] Nov 02 20:19:16.072997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:16.072963 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-7plf7"] Nov 02 20:19:17.083744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.083705 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q"] Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085046 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ae247daf-d610-4e4d-ad75-044103a9291c" containerName="payload-processing" Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085074 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae247daf-d610-4e4d-ad75-044103a9291c" containerName="payload-processing" Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085139 5800 cpu_manager.go:414] "containerMap: removing container" podUID="fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" containerName="payload-pre-processing" Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085147 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" containerName="payload-pre-processing" Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085184 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085195 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085231 5800 cpu_manager.go:414] "containerMap: removing container" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" Nov 02 20:19:17.085340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085237 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" Nov 02 20:19:17.085833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085561 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7bdb31-fb20-4254-be7e-fb70f0c70dfd" containerName="payload-pre-processing" Nov 02 20:19:17.085833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085593 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" Nov 02 20:19:17.085833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085619 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" Nov 02 20:19:17.085833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085639 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" containerName="istio-proxy" Nov 02 20:19:17.085833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.085662 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae247daf-d610-4e4d-ad75-044103a9291c" containerName="payload-processing" Nov 02 20:19:17.092869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.092844 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.182304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.164978 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q"] Nov 02 20:19:17.198152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.198112 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d3184c\"" Nov 02 20:19:17.198385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.198315 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d3184c-dockercfg-ppj8v\"" Nov 02 20:19:17.219343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.218917 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.219343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.219037 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbcb\" (UniqueName: \"kubernetes.io/projected/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-kube-api-access-vdbcb\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.259998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.259960 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc"] Nov 02 20:19:17.260986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.260914 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" Nov 02 20:19:17.260986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.260943 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8398ad9-e519-4c49-a40e-1efedaf04d5d" containerName="istio-proxy" Nov 02 20:19:17.265277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.265253 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.313326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.313272 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc"] Nov 02 20:19:17.320587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.320557 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbcb\" (UniqueName: \"kubernetes.io/projected/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-kube-api-access-vdbcb\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.320770 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.320713 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.323033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.323010 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.376637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.376565 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbcb\" (UniqueName: \"kubernetes.io/projected/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-kube-api-access-vdbcb\") pod \"payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.407119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.407084 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:17.423436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.421795 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97mhl\" (UniqueName: \"kubernetes.io/projected/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-kube-api-access-97mhl\") pod \"payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.423436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.422030 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.524738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.523610 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97mhl\" (UniqueName: \"kubernetes.io/projected/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-kube-api-access-97mhl\") pod \"payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.524738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.523872 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.525504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.525320 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.577786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.577756 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97mhl\" (UniqueName: \"kubernetes.io/projected/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-kube-api-access-97mhl\") pod \"payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.580948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.580922 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:17.625936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.625410 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69834227-930d-4f3b-af29-8e3d93c76e81" path="/var/lib/kubelet/pods/69834227-930d-4f3b-af29-8e3d93c76e81/volumes" Nov 02 20:19:17.805191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.805156 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q"] Nov 02 20:19:17.805436 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:19:17.805409 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f776f90_da6a_4a8b_9213_4b9fd8cc4d0d.slice/crio-833d27cf4f923830fea106b7000352dd647fd04ea97ff530635f8cb99891ea2d WatchSource:0}: Error finding container 833d27cf4f923830fea106b7000352dd647fd04ea97ff530635f8cb99891ea2d: Status 404 returned error can't find the container with id 833d27cf4f923830fea106b7000352dd647fd04ea97ff530635f8cb99891ea2d Nov 02 20:19:17.841001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.840973 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" event={"ID":"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d","Type":"ContainerStarted","Data":"833d27cf4f923830fea106b7000352dd647fd04ea97ff530635f8cb99891ea2d"} Nov 02 20:19:17.959358 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:19:17.959328 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87eaebf_e2ac_4a15_a657_b9ed0a23e7a8.slice/crio-94b990b98dec8b2e4064732e88805c152112f43f2921803eb01a61984d401465 WatchSource:0}: Error finding container 94b990b98dec8b2e4064732e88805c152112f43f2921803eb01a61984d401465: Status 404 returned error can't find the container with id 94b990b98dec8b2e4064732e88805c152112f43f2921803eb01a61984d401465 Nov 02 20:19:17.959755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:17.959729 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc"] Nov 02 20:19:18.851504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:18.851460 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" event={"ID":"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d","Type":"ContainerStarted","Data":"85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9"} Nov 02 20:19:18.852040 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:18.851530 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:18.853243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:18.853210 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" event={"ID":"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8","Type":"ContainerStarted","Data":"f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd"} Nov 02 20:19:18.853399 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:18.853273 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:18.853399 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:18.853332 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" event={"ID":"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8","Type":"ContainerStarted","Data":"94b990b98dec8b2e4064732e88805c152112f43f2921803eb01a61984d401465"} Nov 02 20:19:18.959921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:18.959860 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" podStartSLOduration=1.959842986 podStartE2EDuration="1.959842986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:18.958939854 +0000 UTC m=+1857.475712439" watchObservedRunningTime="2026-11-02 20:19:18.959842986 +0000 UTC m=+1857.476615572" Nov 02 20:19:19.064640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:19.064573 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" podStartSLOduration=2.064553224 podStartE2EDuration="2.064553224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:19.063681404 +0000 UTC m=+1857.580453990" watchObservedRunningTime="2026-11-02 20:19:19.064553224 +0000 UTC m=+1857.581325811" Nov 02 20:19:20.947192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:20.947159 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk"] Nov 02 20:19:20.951671 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:20.951647 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.114856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.114574 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-316e97-openshift-default-dockercfg-fkw4f\"" Nov 02 20:19:21.130344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.130299 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk"] Nov 02 20:19:21.141417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141092 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-data\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141149 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8571aea-4c23-457c-a12c-4cc346a63e4c-istiod-ca-cert\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141205 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np6nm\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-kube-api-access-np6nm\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141250 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-podinfo\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141323 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-envoy\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141417 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-socket\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141446 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-token\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141511 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-certs\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.141859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.141595 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-credential-socket\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.242806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.242715 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-socket\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.242806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.242753 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-token\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.242806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.242802 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-certs\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.242879 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-credential-socket\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.242946 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-data\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.242984 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8571aea-4c23-457c-a12c-4cc346a63e4c-istiod-ca-cert\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.243039 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np6nm\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-kube-api-access-np6nm\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.243089 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-podinfo\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.243119 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-envoy\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.243179 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-socket\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.243485 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-data\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.243790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.243766 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-certs\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.244026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.244006 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-credential-socket\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.244141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.244114 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8571aea-4c23-457c-a12c-4cc346a63e4c-istiod-ca-cert\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.246007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.245958 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-envoy\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.246410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.246388 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-podinfo\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.295804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.295773 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-token\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.295979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.295957 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np6nm\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-kube-api-access-np6nm\") pod \"e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.569972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.569935 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:21.992548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.992522 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk"] Nov 02 20:19:21.993105 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:19:21.993075 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8571aea_4c23_457c_a12c_4cc346a63e4c.slice/crio-6bc9438196f51e39b69fe07716bf238bf70dece7118a6174e9c8afef46f420f0 WatchSource:0}: Error finding container 6bc9438196f51e39b69fe07716bf238bf70dece7118a6174e9c8afef46f420f0: Status 404 returned error can't find the container with id 6bc9438196f51e39b69fe07716bf238bf70dece7118a6174e9c8afef46f420f0 Nov 02 20:19:21.995698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.995660 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:21.995826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.995742 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:21.995826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:21.995788 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:22.401165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:22.401132 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t"] Nov 02 20:19:22.721408 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:22.712913 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:19:22.886542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:22.886503 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" event={"ID":"d8571aea-4c23-457c-a12c-4cc346a63e4c","Type":"ContainerStarted","Data":"33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160"} Nov 02 20:19:22.886692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:22.886615 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" event={"ID":"d8571aea-4c23-457c-a12c-4cc346a63e4c","Type":"ContainerStarted","Data":"6bc9438196f51e39b69fe07716bf238bf70dece7118a6174e9c8afef46f420f0"} Nov 02 20:19:22.998164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:22.998054 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podStartSLOduration=2.998038678 podStartE2EDuration="2.998038678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:22.994860393 +0000 UTC m=+1861.511632977" watchObservedRunningTime="2026-11-02 20:19:22.998038678 +0000 UTC m=+1861.514811256" Nov 02 20:19:23.570697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:23.570664 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:23.576940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:23.576916 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:23.895628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:23.895547 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:23.896647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:23.896628 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:19:27.402223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.402176 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" podUID="9b5c97d6-a151-45e4-979d-7996b8782265" containerName="payload-pre-processing" containerID="cri-o://9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38" gracePeriod=25 Nov 02 20:19:27.692098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.692070 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:19:27.832618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.832578 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5c97d6-a151-45e4-979d-7996b8782265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c97d6-a151-45e4-979d-7996b8782265-plugins-config-volume\") pod \"9b5c97d6-a151-45e4-979d-7996b8782265\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " Nov 02 20:19:27.832956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.832933 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5c97d6-a151-45e4-979d-7996b8782265-kube-api-access-mcmrh\" (UniqueName: \"kubernetes.io/projected/9b5c97d6-a151-45e4-979d-7996b8782265-kube-api-access-mcmrh\") pod \"9b5c97d6-a151-45e4-979d-7996b8782265\" (UID: \"9b5c97d6-a151-45e4-979d-7996b8782265\") " Nov 02 20:19:27.833032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.833005 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5c97d6-a151-45e4-979d-7996b8782265-plugins-config-volume" pod "9b5c97d6-a151-45e4-979d-7996b8782265" (UID: "9b5c97d6-a151-45e4-979d-7996b8782265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:27.833459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.833428 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b5c97d6-a151-45e4-979d-7996b8782265-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:27.835263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.835230 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5c97d6-a151-45e4-979d-7996b8782265-kube-api-access-mcmrh" pod "9b5c97d6-a151-45e4-979d-7996b8782265" (UID: "9b5c97d6-a151-45e4-979d-7996b8782265"). InnerVolumeSpecName "kube-api-access-mcmrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:27.925498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.925408 5800 generic.go:356] "Generic (PLEG): container finished" podID="9b5c97d6-a151-45e4-979d-7996b8782265" containerID="9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38" exitCode=0 Nov 02 20:19:27.925498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.925461 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" Nov 02 20:19:27.925685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.925498 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" event={"ID":"9b5c97d6-a151-45e4-979d-7996b8782265","Type":"ContainerDied","Data":"9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38"} Nov 02 20:19:27.925685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.925570 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t" event={"ID":"9b5c97d6-a151-45e4-979d-7996b8782265","Type":"ContainerDied","Data":"03467f94ddcf47e3daef26e3e821ed1d14d1ba0bef3e6be3d5bf1a75312f022a"} Nov 02 20:19:27.925685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.925599 5800 scope.go:122] "RemoveContainer" containerID="9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38" Nov 02 20:19:27.934724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.934700 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcmrh\" (UniqueName: \"kubernetes.io/projected/9b5c97d6-a151-45e4-979d-7996b8782265-kube-api-access-mcmrh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:27.936017 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.935997 5800 scope.go:122] "RemoveContainer" containerID="9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38" Nov 02 20:19:27.936274 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:27.936255 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38\": container with ID starting with 9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38 not found: ID does not exist" containerID="9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38" Nov 02 20:19:27.936378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:27.936351 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38"} err="failed to get container status \"9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38\": rpc error: code = NotFound desc = could not find container \"9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38\": container with ID starting with 9def9820e338236137cbde6d64024bb2b7787305ba78d3947a1694e2f28b6f38 not found: ID does not exist" Nov 02 20:19:28.080847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:28.080814 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t"] Nov 02 20:19:28.083840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:28.083814 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-f7bdd6c9f-z2j5t"] Nov 02 20:19:29.632633 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:29.632603 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5c97d6-a151-45e4-979d-7996b8782265" path="/var/lib/kubelet/pods/9b5c97d6-a151-45e4-979d-7996b8782265/volumes" Nov 02 20:19:29.887030 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:29.884011 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:19:29.896128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:29.896095 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:19:30.348092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.347226 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp"] Nov 02 20:19:30.348092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.347997 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9b5c97d6-a151-45e4-979d-7996b8782265" containerName="payload-pre-processing" Nov 02 20:19:30.348092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.348013 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5c97d6-a151-45e4-979d-7996b8782265" containerName="payload-pre-processing" Nov 02 20:19:30.348453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.348146 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5c97d6-a151-45e4-979d-7996b8782265" containerName="payload-pre-processing" Nov 02 20:19:30.359113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.359084 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:30.368323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.367927 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp"] Nov 02 20:19:30.408660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.408627 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-316e97-dockercfg-qdbwn\"" Nov 02 20:19:30.408660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.408642 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-316e97\"" Nov 02 20:19:30.491003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.490966 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbf9\" (UniqueName: \"kubernetes.io/projected/9150110d-a61e-4508-876d-bc3d0f8263c0-kube-api-access-lxbf9\") pod \"payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:30.491181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.491165 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9150110d-a61e-4508-876d-bc3d0f8263c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:30.592376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.592338 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9150110d-a61e-4508-876d-bc3d0f8263c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:30.592554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.592460 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbf9\" (UniqueName: \"kubernetes.io/projected/9150110d-a61e-4508-876d-bc3d0f8263c0-kube-api-access-lxbf9\") pod \"payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:30.593060 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.593041 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9150110d-a61e-4508-876d-bc3d0f8263c0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:30.645438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.645360 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbf9\" (UniqueName: \"kubernetes.io/projected/9150110d-a61e-4508-876d-bc3d0f8263c0-kube-api-access-lxbf9\") pod \"payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:30.684157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:30.684125 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:31.058588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:31.058563 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp"] Nov 02 20:19:31.059622 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:19:31.059595 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9150110d_a61e_4508_876d_bc3d0f8263c0.slice/crio-77d62876b25e71b9fa7bc52e30f23db2e5b4800c62c30c92527b8c040cbd35bd WatchSource:0}: Error finding container 77d62876b25e71b9fa7bc52e30f23db2e5b4800c62c30c92527b8c040cbd35bd: Status 404 returned error can't find the container with id 77d62876b25e71b9fa7bc52e30f23db2e5b4800c62c30c92527b8c040cbd35bd Nov 02 20:19:31.955250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:31.955211 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" event={"ID":"9150110d-a61e-4508-876d-bc3d0f8263c0","Type":"ContainerStarted","Data":"4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf"} Nov 02 20:19:31.955669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:31.955272 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" event={"ID":"9150110d-a61e-4508-876d-bc3d0f8263c0","Type":"ContainerStarted","Data":"77d62876b25e71b9fa7bc52e30f23db2e5b4800c62c30c92527b8c040cbd35bd"} Nov 02 20:19:31.955669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:31.955315 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:32.061333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:32.061251 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" podStartSLOduration=2.061233667 podStartE2EDuration="2.061233667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:32.059569275 +0000 UTC m=+1870.576341861" watchObservedRunningTime="2026-11-02 20:19:32.061233667 +0000 UTC m=+1870.578006254" Nov 02 20:19:33.047199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:33.047162 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8"] Nov 02 20:19:33.089592 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:33.089547 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22cda722-vzhc8"] Nov 02 20:19:33.622161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:33.622129 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab4c922-3941-4913-ab35-a35022eea89f" path="/var/lib/kubelet/pods/7ab4c922-3941-4913-ab35-a35022eea89f/volumes" Nov 02 20:19:38.727392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:38.727353 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w"] Nov 02 20:19:38.728356 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:38.728317 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" containerID="cri-o://bc832d99da7b1a01ea4476292795053e47431339d37255c31baa775cc9884d51" gracePeriod=30 Nov 02 20:19:38.732083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:38.732063 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt"] Nov 02 20:19:38.732471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:38.732419 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" containerID="cri-o://8dd226cffdf299c9eadbf67b5a2c956bc37ba9f45dd7dbda2d0f84d2afb6eda9" gracePeriod=30 Nov 02 20:19:38.769786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:38.769754 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn"] Nov 02 20:19:38.769969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:38.769935 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" containerID="cri-o://44752c2ee52358efece054d8b8f721d4f25efc3f42f4d2b94406a255ea97a174" gracePeriod=30 Nov 02 20:19:40.377104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.377063 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx"] Nov 02 20:19:40.390583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.390558 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:40.433003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.432964 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx"] Nov 02 20:19:40.452651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.452612 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:40.452879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.452761 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6r6\" (UniqueName: \"kubernetes.io/projected/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-kube-api-access-cf6r6\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:40.475232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.475193 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f83bc2\"" Nov 02 20:19:40.512716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.512685 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f83bc2-dockercfg-x466x\"" Nov 02 20:19:40.554768 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.554726 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:40.554951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.554822 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6r6\" (UniqueName: \"kubernetes.io/projected/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-kube-api-access-cf6r6\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:40.555575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.555553 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:40.607995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.607953 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6r6\" (UniqueName: \"kubernetes.io/projected/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-kube-api-access-cf6r6\") pod \"payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:40.708231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:40.708124 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:41.081375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.081348 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx"] Nov 02 20:19:41.082701 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:19:41.082669 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3e3d07_9a3f_4cd7_8363_9cbf30621371.slice/crio-fd80bfd54ee4c6fb3ca84166b0b3f607890af1789d48926422260ec6bb9d9818 WatchSource:0}: Error finding container fd80bfd54ee4c6fb3ca84166b0b3f607890af1789d48926422260ec6bb9d9818: Status 404 returned error can't find the container with id fd80bfd54ee4c6fb3ca84166b0b3f607890af1789d48926422260ec6bb9d9818 Nov 02 20:19:41.084980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.084959 5800 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:19:41.125701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.125671 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z"] Nov 02 20:19:41.131316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.131271 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.163683 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.163654 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.163880 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.163862 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4dp\" (UniqueName: \"kubernetes.io/projected/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-kube-api-access-dh4dp\") pod \"payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.182181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.182155 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z"] Nov 02 20:19:41.265643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.265606 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.265820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.265782 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4dp\" (UniqueName: \"kubernetes.io/projected/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-kube-api-access-dh4dp\") pod \"payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.266381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.266359 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.318703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.318678 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4dp\" (UniqueName: \"kubernetes.io/projected/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-kube-api-access-dh4dp\") pod \"payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.446556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.446474 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:41.819662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:41.819618 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z"] Nov 02 20:19:42.029772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:42.029736 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" event={"ID":"5b7169de-a109-4025-8ce8-af8bdcbaa1fa","Type":"ContainerStarted","Data":"6c2057a28aa15f6905149a35d519c83d0ec329608a17a0b60d02eab5476a7618"} Nov 02 20:19:42.031113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:42.031085 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" event={"ID":"ca3e3d07-9a3f-4cd7-8363-9cbf30621371","Type":"ContainerStarted","Data":"fd80bfd54ee4c6fb3ca84166b0b3f607890af1789d48926422260ec6bb9d9818"} Nov 02 20:19:42.967390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:42.967357 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:19:44.055061 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.054936 5800 generic.go:356] "Generic (PLEG): container finished" podID="6397d570-9ef1-47ca-a927-c397d174fe67" containerID="bc832d99da7b1a01ea4476292795053e47431339d37255c31baa775cc9884d51" exitCode=0 Nov 02 20:19:44.055525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.055124 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" event={"ID":"6397d570-9ef1-47ca-a927-c397d174fe67","Type":"ContainerDied","Data":"bc832d99da7b1a01ea4476292795053e47431339d37255c31baa775cc9884d51"} Nov 02 20:19:44.055525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.055180 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" event={"ID":"6397d570-9ef1-47ca-a927-c397d174fe67","Type":"ContainerDied","Data":"604fa499f0310b10c51cc84f67f3559bd970ea4a79979d48a876f6b6ee4f4d07"} Nov 02 20:19:44.055525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.055207 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604fa499f0310b10c51cc84f67f3559bd970ea4a79979d48a876f6b6ee4f4d07" Nov 02 20:19:44.058040 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.058017 5800 generic.go:356] "Generic (PLEG): container finished" podID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerID="8dd226cffdf299c9eadbf67b5a2c956bc37ba9f45dd7dbda2d0f84d2afb6eda9" exitCode=0 Nov 02 20:19:44.058179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.058162 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" event={"ID":"585fcd70-8a96-4a96-88b7-f088fad23b01","Type":"ContainerDied","Data":"8dd226cffdf299c9eadbf67b5a2c956bc37ba9f45dd7dbda2d0f84d2afb6eda9"} Nov 02 20:19:44.058252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.058204 5800 scope.go:122] "RemoveContainer" containerID="7d25e5ad3c51b0a42f00bf0fad37c678fe54100722fcf92729fce5e792ba2dc3" Nov 02 20:19:44.062539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.062507 5800 generic.go:356] "Generic (PLEG): container finished" podID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerID="44752c2ee52358efece054d8b8f721d4f25efc3f42f4d2b94406a255ea97a174" exitCode=0 Nov 02 20:19:44.062653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.062579 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" event={"ID":"9fc22d1f-0707-40dd-9527-6f88e8b89a7c","Type":"ContainerDied","Data":"44752c2ee52358efece054d8b8f721d4f25efc3f42f4d2b94406a255ea97a174"} Nov 02 20:19:44.076326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.075465 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:19:44.103378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103348 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-certs\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.103521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103426 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-data\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.103521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103469 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-socket\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.103660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103616 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-certs" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.103844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103822 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-data" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.103982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103920 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-envoy\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.103982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103933 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-socket" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.103982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.103977 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6397d570-9ef1-47ca-a927-c397d174fe67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6397d570-9ef1-47ca-a927-c397d174fe67-istio-podinfo\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.104590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104014 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-kube-api-access-h84zx\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-kube-api-access-h84zx\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.104590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104056 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-credential-socket\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.104590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104256 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6397d570-9ef1-47ca-a927-c397d174fe67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6397d570-9ef1-47ca-a927-c397d174fe67-istiod-ca-cert\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.104590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104328 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-istio-token\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-istio-token\") pod \"6397d570-9ef1-47ca-a927-c397d174fe67\" (UID: \"6397d570-9ef1-47ca-a927-c397d174fe67\") " Nov 02 20:19:44.105048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104862 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-credential-socket" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.105048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104878 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.105048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104900 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.105048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.104916 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.105483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.105339 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:19:44.105742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.105722 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6397d570-9ef1-47ca-a927-c397d174fe67-istiod-ca-cert" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:44.106628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.106594 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-envoy" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.111709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.111680 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6397d570-9ef1-47ca-a927-c397d174fe67-istio-podinfo" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:44.114556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.114482 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-kube-api-access-h84zx" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "kube-api-access-h84zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.114556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.114528 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-istio-token" pod "6397d570-9ef1-47ca-a927-c397d174fe67" (UID: "6397d570-9ef1-47ca-a927-c397d174fe67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.139650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.139625 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:19:44.205786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.205755 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-socket\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.205935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.205811 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-kube-api-access-9t2nf\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-kube-api-access-9t2nf\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.205935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.205878 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-credential-socket\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.206034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.205946 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-envoy\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.206034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.205988 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-podinfo\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.206034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206028 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-token\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206053 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585fcd70-8a96-4a96-88b7-f088fad23b01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585fcd70-8a96-4a96-88b7-f088fad23b01-istiod-ca-cert\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.206204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206077 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-socket" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.206204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206095 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-token\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-token\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.206204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206189 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-data\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206202 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-credential-socket" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.206486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206233 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-certs\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206314 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-certs\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.206486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206381 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-kube-api-access-gbr72\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-kube-api-access-gbr72\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.206486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206409 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-podinfo\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206498 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-envoy\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206534 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-credential-socket\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206568 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585fcd70-8a96-4a96-88b7-f088fad23b01-istiod-ca-cert" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:44.206743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206578 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istiod-ca-cert\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206583 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-data" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.206743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206627 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-socket\") pod \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\" (UID: \"9fc22d1f-0707-40dd-9527-6f88e8b89a7c\") " Nov 02 20:19:44.206743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206665 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-data\") pod \"585fcd70-8a96-4a96-88b7-f088fad23b01\" (UID: \"585fcd70-8a96-4a96-88b7-f088fad23b01\") " Nov 02 20:19:44.207122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206788 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-certs" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.207122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.206968 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-certs" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.207237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207172 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-data" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.207403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207381 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-credential-socket" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.207488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207436 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207459 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207474 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6397d570-9ef1-47ca-a927-c397d174fe67-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207494 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84zx\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-kube-api-access-h84zx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207509 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6397d570-9ef1-47ca-a927-c397d174fe67-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207525 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585fcd70-8a96-4a96-88b7-f088fad23b01-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207540 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207554 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207572 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207588 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6397d570-9ef1-47ca-a927-c397d174fe67-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207602 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6397d570-9ef1-47ca-a927-c397d174fe67-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207617 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.207659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207630 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.208185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.207957 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istiod-ca-cert" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:19:44.208248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.208197 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-socket" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.208834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.208806 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-podinfo" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:44.208956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.208847 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-token" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.208956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.208879 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-token" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.209726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.209693 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-kube-api-access-gbr72" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "kube-api-access-gbr72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.210109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.210085 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-envoy" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.210312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.210273 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-kube-api-access-9t2nf" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "kube-api-access-9t2nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:19:44.210761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.210741 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-podinfo" pod "9fc22d1f-0707-40dd-9527-6f88e8b89a7c" (UID: "9fc22d1f-0707-40dd-9527-6f88e8b89a7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:19:44.211053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.211036 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-envoy" pod "585fcd70-8a96-4a96-88b7-f088fad23b01" (UID: "585fcd70-8a96-4a96-88b7-f088fad23b01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:19:44.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308889 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308918 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308932 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2nf\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-kube-api-access-9t2nf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308942 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308952 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308962 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.309264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308971 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.309264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308980 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbr72\" (UniqueName: \"kubernetes.io/projected/585fcd70-8a96-4a96-88b7-f088fad23b01-kube-api-access-gbr72\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.309264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308989 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.309264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.308997 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:44.309264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:44.309006 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc22d1f-0707-40dd-9527-6f88e8b89a7c-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:19:45.073119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.073029 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" Nov 02 20:19:45.073119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.073058 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt" event={"ID":"585fcd70-8a96-4a96-88b7-f088fad23b01","Type":"ContainerDied","Data":"2acf0120825f6513623a56b0a7952d58e7917d57653d5d09efb0683231f7b424"} Nov 02 20:19:45.073671 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.073129 5800 scope.go:122] "RemoveContainer" containerID="8dd226cffdf299c9eadbf67b5a2c956bc37ba9f45dd7dbda2d0f84d2afb6eda9" Nov 02 20:19:45.077784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.077739 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" Nov 02 20:19:45.077963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.077795 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn" event={"ID":"9fc22d1f-0707-40dd-9527-6f88e8b89a7c","Type":"ContainerDied","Data":"6c382186dc015c9d1b29f403bae74aa87463aa9448faa352068e380e8ecb9fc7"} Nov 02 20:19:45.078480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.077984 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w" Nov 02 20:19:45.094458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.094436 5800 scope.go:122] "RemoveContainer" containerID="44752c2ee52358efece054d8b8f721d4f25efc3f42f4d2b94406a255ea97a174" Nov 02 20:19:45.230818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.230713 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt"] Nov 02 20:19:45.274614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.274575 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-4mrzt"] Nov 02 20:19:45.337694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.337596 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn"] Nov 02 20:19:45.342410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.342380 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-v95fn"] Nov 02 20:19:45.444545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.444509 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w"] Nov 02 20:19:45.448638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.448612 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-40aecd-openshift-default-5d8dc5f56f-hgg7w"] Nov 02 20:19:45.625778 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.625696 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" path="/var/lib/kubelet/pods/585fcd70-8a96-4a96-88b7-f088fad23b01/volumes" Nov 02 20:19:45.626329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.626310 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" path="/var/lib/kubelet/pods/6397d570-9ef1-47ca-a927-c397d174fe67/volumes" Nov 02 20:19:45.626766 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:45.626751 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" path="/var/lib/kubelet/pods/9fc22d1f-0707-40dd-9527-6f88e8b89a7c/volumes" Nov 02 20:19:47.097115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:47.097069 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" event={"ID":"ca3e3d07-9a3f-4cd7-8363-9cbf30621371","Type":"ContainerStarted","Data":"e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00"} Nov 02 20:19:47.097574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:47.097144 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:47.098837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:47.098814 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" event={"ID":"5b7169de-a109-4025-8ce8-af8bdcbaa1fa","Type":"ContainerStarted","Data":"6332595fbb6fd1b800f8dc7842784f54874330d7d851d12c63bf56ac7f0ce263"} Nov 02 20:19:47.098969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:47.098949 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:19:47.206216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:47.206159 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" podStartSLOduration=1.3041255569999999 podStartE2EDuration="7.206142925s" totalImagesPullingTime="5.902017368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:47.202712391 +0000 UTC m=+1885.719484976" watchObservedRunningTime="2026-11-02 20:19:47.206142925 +0000 UTC m=+1885.722915510" Nov 02 20:19:47.309103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:47.309036 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" podStartSLOduration=1.444980728 podStartE2EDuration="6.309018505s" totalImagesPullingTime="4.864037777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:47.308244669 +0000 UTC m=+1885.825017249" watchObservedRunningTime="2026-11-02 20:19:47.309018505 +0000 UTC m=+1885.825791091" Nov 02 20:19:52.772141 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:19:52.772108 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.029512 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr"] Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.030944 5800 cpu_manager.go:414] "containerMap: removing container" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.030967 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031002 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031011 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031042 5800 cpu_manager.go:414] "containerMap: removing container" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031049 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031108 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031115 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031569 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031599 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc22d1f-0707-40dd-9527-6f88e8b89a7c" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031621 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="585fcd70-8a96-4a96-88b7-f088fad23b01" containerName="istio-proxy" Nov 02 20:19:54.035758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.031645 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6397d570-9ef1-47ca-a927-c397d174fe67" containerName="istio-proxy" Nov 02 20:19:54.041042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.041009 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.086637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.086601 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr"] Nov 02 20:19:54.125486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.125443 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6c9cce-ee3f-4650-842c-d800b7051c33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.125691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.125497 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqrc\" (UniqueName: \"kubernetes.io/projected/6b6c9cce-ee3f-4650-842c-d800b7051c33-kube-api-access-dwqrc\") pod \"payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.129110 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.129074 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a999d7\"" Nov 02 20:19:54.129110 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.129075 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a999d7-dockercfg-mt4zc\"" Nov 02 20:19:54.226749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.226715 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6c9cce-ee3f-4650-842c-d800b7051c33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.226936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.226766 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqrc\" (UniqueName: \"kubernetes.io/projected/6b6c9cce-ee3f-4650-842c-d800b7051c33-kube-api-access-dwqrc\") pod \"payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.227550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.227532 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6c9cce-ee3f-4650-842c-d800b7051c33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.279781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.279750 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqrc\" (UniqueName: \"kubernetes.io/projected/6b6c9cce-ee3f-4650-842c-d800b7051c33-kube-api-access-dwqrc\") pod \"payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.355379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.355261 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:54.728660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:54.728561 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr"] Nov 02 20:19:54.730211 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:19:54.730183 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6c9cce_ee3f_4650_842c_d800b7051c33.slice/crio-00ba27ea618c13dada199266c37540bf6d9dd763b26410434c5481ba570a2cb3 WatchSource:0}: Error finding container 00ba27ea618c13dada199266c37540bf6d9dd763b26410434c5481ba570a2cb3: Status 404 returned error can't find the container with id 00ba27ea618c13dada199266c37540bf6d9dd763b26410434c5481ba570a2cb3 Nov 02 20:19:55.222252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:55.222189 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" event={"ID":"6b6c9cce-ee3f-4650-842c-d800b7051c33","Type":"ContainerStarted","Data":"d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7"} Nov 02 20:19:55.222787 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:55.222581 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:19:55.222787 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:55.222645 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" event={"ID":"6b6c9cce-ee3f-4650-842c-d800b7051c33","Type":"ContainerStarted","Data":"00ba27ea618c13dada199266c37540bf6d9dd763b26410434c5481ba570a2cb3"} Nov 02 20:19:55.329058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:55.328994 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" podStartSLOduration=2.328979912 podStartE2EDuration="2.328979912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:55.327406457 +0000 UTC m=+1893.844179046" watchObservedRunningTime="2026-11-02 20:19:55.328979912 +0000 UTC m=+1893.845752497" Nov 02 20:19:58.108465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:58.108427 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:19:58.108868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:19:58.108522 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:20:06.232269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:06.232184 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:20:16.169810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.169764 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh"] Nov 02 20:20:16.174951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.174921 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.224055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.224023 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-296ae9\"" Nov 02 20:20:16.226986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.226946 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh"] Nov 02 20:20:16.252165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.252115 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-296ae9-dockercfg-94g4m\"" Nov 02 20:20:16.324695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.324643 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7h6\" (UniqueName: \"kubernetes.io/projected/f09d0560-6c7d-47ce-8015-ca0446aff452-kube-api-access-pp7h6\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.324934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.324905 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09d0560-6c7d-47ce-8015-ca0446aff452-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.426112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.426001 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7h6\" (UniqueName: \"kubernetes.io/projected/f09d0560-6c7d-47ce-8015-ca0446aff452-kube-api-access-pp7h6\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.426320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.426231 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09d0560-6c7d-47ce-8015-ca0446aff452-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.427004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.426973 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09d0560-6c7d-47ce-8015-ca0446aff452-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.479704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.479670 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7h6\" (UniqueName: \"kubernetes.io/projected/f09d0560-6c7d-47ce-8015-ca0446aff452-kube-api-access-pp7h6\") pod \"payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.490380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.490324 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:16.867606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:16.867564 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh"] Nov 02 20:20:16.868439 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:20:16.868391 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09d0560_6c7d_47ce_8015_ca0446aff452.slice/crio-4a8825996057c6181764a40a4929ae07ec21b800ec4a562922b4643d038dfeb8 WatchSource:0}: Error finding container 4a8825996057c6181764a40a4929ae07ec21b800ec4a562922b4643d038dfeb8: Status 404 returned error can't find the container with id 4a8825996057c6181764a40a4929ae07ec21b800ec4a562922b4643d038dfeb8 Nov 02 20:20:17.412166 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:17.412127 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" event={"ID":"f09d0560-6c7d-47ce-8015-ca0446aff452","Type":"ContainerStarted","Data":"ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362"} Nov 02 20:20:17.412644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:17.412188 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" event={"ID":"f09d0560-6c7d-47ce-8015-ca0446aff452","Type":"ContainerStarted","Data":"4a8825996057c6181764a40a4929ae07ec21b800ec4a562922b4643d038dfeb8"} Nov 02 20:20:17.412644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:17.412314 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:17.520526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:17.520466 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" podStartSLOduration=1.520451043 podStartE2EDuration="1.520451043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:17.517621056 +0000 UTC m=+1916.034393637" watchObservedRunningTime="2026-11-02 20:20:17.520451043 +0000 UTC m=+1916.037223619" Nov 02 20:20:18.289224 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.289182 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn"] Nov 02 20:20:18.294468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.294441 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.348303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.344660 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:20:18.354565 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.354503 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.354758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.354612 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.354837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.354822 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r55s\" (UniqueName: \"kubernetes.io/projected/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-kube-api-access-9r55s\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.390989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.390957 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:20:18.394565 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.394532 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn"] Nov 02 20:20:18.422096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.422044 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:20:18.422529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.422143 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:20:18.458758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.457556 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r55s\" (UniqueName: \"kubernetes.io/projected/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-kube-api-access-9r55s\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.458758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.457720 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.458758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.457758 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.460803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.459996 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.463023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.462987 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.512362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.512326 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r55s\" (UniqueName: \"kubernetes.io/projected/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-kube-api-access-9r55s\") pod \"maas-api-revoke-sub-d405494c-f2kdn\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.610636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.610546 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:18.994436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:18.994400 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn"] Nov 02 20:20:18.995553 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:20:18.995527 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9f1d7c_50c7_49c2_b909_0e76b04414a3.slice/crio-c7de720b17d3430064afb94bb09a5884bf5d132e0de0598cdbaa752584bc1747 WatchSource:0}: Error finding container c7de720b17d3430064afb94bb09a5884bf5d132e0de0598cdbaa752584bc1747: Status 404 returned error can't find the container with id c7de720b17d3430064afb94bb09a5884bf5d132e0de0598cdbaa752584bc1747 Nov 02 20:20:19.430638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:19.430597 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" event={"ID":"4c9f1d7c-50c7-49c2-b909-0e76b04414a3","Type":"ContainerStarted","Data":"304070b0a7d532e738e6834e72ff8104226d92f769e75bc2d163ff9a70d208de"} Nov 02 20:20:19.431163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:19.430657 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" event={"ID":"4c9f1d7c-50c7-49c2-b909-0e76b04414a3","Type":"ContainerStarted","Data":"c7de720b17d3430064afb94bb09a5884bf5d132e0de0598cdbaa752584bc1747"} Nov 02 20:20:19.540139 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:19.540078 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" podStartSLOduration=1.54006065 podStartE2EDuration="1.54006065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:19.537519576 +0000 UTC m=+1918.054292160" watchObservedRunningTime="2026-11-02 20:20:19.54006065 +0000 UTC m=+1918.056833234" Nov 02 20:20:20.440607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:20.440564 5800 generic.go:356] "Generic (PLEG): container finished" podID="4c9f1d7c-50c7-49c2-b909-0e76b04414a3" containerID="304070b0a7d532e738e6834e72ff8104226d92f769e75bc2d163ff9a70d208de" exitCode=0 Nov 02 20:20:20.441027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:20.440645 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" event={"ID":"4c9f1d7c-50c7-49c2-b909-0e76b04414a3","Type":"ContainerDied","Data":"304070b0a7d532e738e6834e72ff8104226d92f769e75bc2d163ff9a70d208de"} Nov 02 20:20:21.634506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.634466 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:21.701864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.701821 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-openshift-ca\") pod \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " Nov 02 20:20:21.702101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.701898 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-kube-api-access-9r55s\" (UniqueName: \"kubernetes.io/projected/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-kube-api-access-9r55s\") pod \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " Nov 02 20:20:21.702101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.702015 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-xks-ca\") pod \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\" (UID: \"4c9f1d7c-50c7-49c2-b909-0e76b04414a3\") " Nov 02 20:20:21.702338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.702311 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-openshift-ca" pod "4c9f1d7c-50c7-49c2-b909-0e76b04414a3" (UID: "4c9f1d7c-50c7-49c2-b909-0e76b04414a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:20:21.702642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.702623 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:21.704401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.704340 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-kube-api-access-9r55s" pod "4c9f1d7c-50c7-49c2-b909-0e76b04414a3" (UID: "4c9f1d7c-50c7-49c2-b909-0e76b04414a3"). InnerVolumeSpecName "kube-api-access-9r55s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:21.704574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.704550 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-xks-ca" pod "4c9f1d7c-50c7-49c2-b909-0e76b04414a3" (UID: "4c9f1d7c-50c7-49c2-b909-0e76b04414a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:20:21.803783 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.803744 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:21.803783 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:21.803779 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r55s\" (UniqueName: \"kubernetes.io/projected/4c9f1d7c-50c7-49c2-b909-0e76b04414a3-kube-api-access-9r55s\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:22.463545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:22.463512 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" Nov 02 20:20:22.463769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:22.463511 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn" event={"ID":"4c9f1d7c-50c7-49c2-b909-0e76b04414a3","Type":"ContainerDied","Data":"c7de720b17d3430064afb94bb09a5884bf5d132e0de0598cdbaa752584bc1747"} Nov 02 20:20:22.463769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:22.463635 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7de720b17d3430064afb94bb09a5884bf5d132e0de0598cdbaa752584bc1747" Nov 02 20:20:22.705965 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:20:22.705921 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:20:28.423357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:28.423265 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:20:48.496309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:48.495396 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:20:48.628455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:48.628397 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:20:48.662358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:48.661457 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:20:48.705313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:48.704382 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded" Nov 02 20:20:48.747084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:48.745404 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:20:52.699696 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:20:52.699658 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:20:54.936510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:54.936452 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:20:54.936961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:54.936530 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:20:57.855803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:57.855770 5800 generic.go:356] "Generic (PLEG): container finished" podID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerID="33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160" exitCode=0 Nov 02 20:20:57.856276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:57.855850 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" event={"ID":"d8571aea-4c23-457c-a12c-4cc346a63e4c","Type":"ContainerDied","Data":"33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160"} Nov 02 20:20:57.856393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:57.856322 5800 scope.go:122] "RemoveContainer" containerID="33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160" Nov 02 20:20:57.858044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:57.857988 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:57.858154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:57.858062 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:57.858154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:57.858120 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:20:58.869027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:58.868980 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" event={"ID":"d8571aea-4c23-457c-a12c-4cc346a63e4c","Type":"ContainerStarted","Data":"7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1"} Nov 02 20:20:59.570233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:59.570196 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:20:59.570468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:59.570392 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:20:59.570468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:20:59.570450 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:21:00.570520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:00.570482 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:21:00.571129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:00.571097 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:21:01.570581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:01.570544 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:21:01.571134 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:01.571096 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:21:02.570433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:02.570388 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:21:02.570648 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:02.570442 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:21:03.571270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:03.571228 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 02 20:21:03.571791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:03.571311 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 02 20:21:04.575616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:04.575585 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:21:04.576112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:04.576100 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:21:04.576832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:04.576808 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:21:20.623731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:20.623695 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc"] Nov 02 20:21:20.723831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:20.723801 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q"] Nov 02 20:21:22.718716 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:21:22.718680 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:21:22.922712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:22.922668 5800 scope.go:122] "RemoveContainer" containerID="7de0c920c2b4ae2597167870bccdf2fcc0d43cc752a8a5b1fbf33db94600f01c" Nov 02 20:21:25.625357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:25.625319 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" podUID="a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" containerName="payload-processing" containerID="cri-o://f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd" gracePeriod=25 Nov 02 20:21:25.724294 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:25.724212 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" podUID="7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" containerName="payload-pre-processing" containerID="cri-o://85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9" gracePeriod=25 Nov 02 20:21:25.766400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:25.766373 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:21:25.766498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:25.766438 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:21:25.910870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:25.910833 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:21:26.019528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.019499 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-kube-api-access-97mhl\" (UniqueName: \"kubernetes.io/projected/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-kube-api-access-97mhl\") pod \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " Nov 02 20:21:26.019697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.019564 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-plugins-config-volume\") pod \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\" (UID: \"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8\") " Nov 02 20:21:26.019994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.019963 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-plugins-config-volume" pod "a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" (UID: "a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:26.022479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.022445 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-kube-api-access-97mhl" pod "a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" (UID: "a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8"). InnerVolumeSpecName "kube-api-access-97mhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:26.025256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.025233 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:21:26.111119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.111083 5800 generic.go:356] "Generic (PLEG): container finished" podID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerID="1d1f1d11b9c1a7b4dea4602eac0a8495fd7a077c9a8da1217d5608beb5d3e49c" exitCode=0 Nov 02 20:21:26.111327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.111159 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" event={"ID":"e60d3f55-ff2f-4876-8a0a-3a88340cefe1","Type":"ContainerDied","Data":"1d1f1d11b9c1a7b4dea4602eac0a8495fd7a077c9a8da1217d5608beb5d3e49c"} Nov 02 20:21:26.111667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.111647 5800 scope.go:122] "RemoveContainer" containerID="1d1f1d11b9c1a7b4dea4602eac0a8495fd7a077c9a8da1217d5608beb5d3e49c" Nov 02 20:21:26.112682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.112660 5800 generic.go:356] "Generic (PLEG): container finished" podID="7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" containerID="85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9" exitCode=0 Nov 02 20:21:26.112814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.112721 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" Nov 02 20:21:26.112814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.112772 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" event={"ID":"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d","Type":"ContainerDied","Data":"85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9"} Nov 02 20:21:26.112814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.112809 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q" event={"ID":"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d","Type":"ContainerDied","Data":"833d27cf4f923830fea106b7000352dd647fd04ea97ff530635f8cb99891ea2d"} Nov 02 20:21:26.112986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.112837 5800 scope.go:122] "RemoveContainer" containerID="85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9" Nov 02 20:21:26.113407 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.113381 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:21:26.113490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.113445 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:21:26.113539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.113490 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:21:26.114393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.114367 5800 generic.go:356] "Generic (PLEG): container finished" podID="a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" containerID="f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd" exitCode=0 Nov 02 20:21:26.114490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.114417 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" Nov 02 20:21:26.114490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.114442 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" event={"ID":"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8","Type":"ContainerDied","Data":"f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd"} Nov 02 20:21:26.114611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.114538 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc" event={"ID":"a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8","Type":"ContainerDied","Data":"94b990b98dec8b2e4064732e88805c152112f43f2921803eb01a61984d401465"} Nov 02 20:21:26.120606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.120584 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-kube-api-access-vdbcb\" (UniqueName: \"kubernetes.io/projected/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-kube-api-access-vdbcb\") pod \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " Nov 02 20:21:26.120713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.120642 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-plugins-config-volume\") pod \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\" (UID: \"7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d\") " Nov 02 20:21:26.120971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.120950 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97mhl\" (UniqueName: \"kubernetes.io/projected/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-kube-api-access-97mhl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.121052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.120979 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.121052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.121035 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-plugins-config-volume" pod "7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" (UID: "7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:26.123398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.123373 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-kube-api-access-vdbcb" pod "7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" (UID: "7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d"). InnerVolumeSpecName "kube-api-access-vdbcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:26.128293 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.128266 5800 scope.go:122] "RemoveContainer" containerID="85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9" Nov 02 20:21:26.128613 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:21:26.128590 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9\": container with ID starting with 85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9 not found: ID does not exist" containerID="85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9" Nov 02 20:21:26.128665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.128626 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9"} err="failed to get container status \"85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9\": rpc error: code = NotFound desc = could not find container \"85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9\": container with ID starting with 85b1dcfa05a5f0000bbd285402641679011b011759f1ee68377f31e34c1748c9 not found: ID does not exist" Nov 02 20:21:26.128665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.128649 5800 scope.go:122] "RemoveContainer" containerID="f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd" Nov 02 20:21:26.142597 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.142574 5800 scope.go:122] "RemoveContainer" containerID="f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd" Nov 02 20:21:26.142961 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:21:26.142917 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd\": container with ID starting with f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd not found: ID does not exist" containerID="f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd" Nov 02 20:21:26.143068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.142973 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd"} err="failed to get container status \"f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd\": rpc error: code = NotFound desc = could not find container \"f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd\": container with ID starting with f586ad9cd286505b6c9247c3dd127c785fb4521eaa0c6a764779cadf331058fd not found: ID does not exist" Nov 02 20:21:26.222223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.222180 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.222223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.222218 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdbcb\" (UniqueName: \"kubernetes.io/projected/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d-kube-api-access-vdbcb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:26.240403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.240367 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc"] Nov 02 20:21:26.243352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.243325 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3184c-7975cbf57c-gdhfc"] Nov 02 20:21:26.531867 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.531836 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q"] Nov 02 20:21:26.535190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:26.535166 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d3184c-6c56c77587-ff48q"] Nov 02 20:21:27.123458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:27.123418 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" event={"ID":"e60d3f55-ff2f-4876-8a0a-3a88340cefe1","Type":"ContainerStarted","Data":"2fc0fbd7acf0a816f780999ace61bbde08905aa57859912a74b2ce81a3169e95"} Nov 02 20:21:27.255707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:27.255671 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:21:27.256025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:27.256005 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:21:27.256204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:27.256181 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:21:27.638305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:27.625362 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" path="/var/lib/kubelet/pods/7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d/volumes" Nov 02 20:21:27.639272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:27.639240 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" path="/var/lib/kubelet/pods/a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8/volumes" Nov 02 20:21:28.237305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:28.234691 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:21:28.237305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:28.234774 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:21:29.235305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:29.234656 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:21:29.235305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:29.234720 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:21:30.245773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:30.245680 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:21:30.246365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:30.246331 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:21:31.252746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:31.245662 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:21:31.252746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:31.245728 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:21:32.229596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:32.229556 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 02 20:21:32.229807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:32.229623 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 02 20:21:33.235088 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:33.235059 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:21:33.235618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:33.235548 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:21:33.236222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:33.236201 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:21:48.043874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:48.043836 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5"] Nov 02 20:21:48.089860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:48.089827 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cf3860-8q7k5"] Nov 02 20:21:48.852032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:48.851999 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp"] Nov 02 20:21:49.622728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:49.622670 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c179dd9-0e57-475c-9fa9-a1e0ceff2e10" path="/var/lib/kubelet/pods/9c179dd9-0e57-475c-9fa9-a1e0ceff2e10/volumes" Nov 02 20:21:52.696847 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:21:52.696810 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:21:53.853083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:53.853038 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" podUID="9150110d-a61e-4508-876d-bc3d0f8263c0" containerName="payload-processing" containerID="cri-o://4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf" gracePeriod=25 Nov 02 20:21:54.132629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.132600 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:21:54.204538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.204509 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9150110d-a61e-4508-876d-bc3d0f8263c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9150110d-a61e-4508-876d-bc3d0f8263c0-plugins-config-volume\") pod \"9150110d-a61e-4508-876d-bc3d0f8263c0\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " Nov 02 20:21:54.204767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.204752 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9150110d-a61e-4508-876d-bc3d0f8263c0-kube-api-access-lxbf9\" (UniqueName: \"kubernetes.io/projected/9150110d-a61e-4508-876d-bc3d0f8263c0-kube-api-access-lxbf9\") pod \"9150110d-a61e-4508-876d-bc3d0f8263c0\" (UID: \"9150110d-a61e-4508-876d-bc3d0f8263c0\") " Nov 02 20:21:54.204877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.204852 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9150110d-a61e-4508-876d-bc3d0f8263c0-plugins-config-volume" pod "9150110d-a61e-4508-876d-bc3d0f8263c0" (UID: "9150110d-a61e-4508-876d-bc3d0f8263c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:21:54.205174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.205157 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9150110d-a61e-4508-876d-bc3d0f8263c0-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:54.206905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.206877 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9150110d-a61e-4508-876d-bc3d0f8263c0-kube-api-access-lxbf9" pod "9150110d-a61e-4508-876d-bc3d0f8263c0" (UID: "9150110d-a61e-4508-876d-bc3d0f8263c0"). InnerVolumeSpecName "kube-api-access-lxbf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:21:54.306164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.306131 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbf9\" (UniqueName: \"kubernetes.io/projected/9150110d-a61e-4508-876d-bc3d0f8263c0-kube-api-access-lxbf9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:21:54.363480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.363380 5800 generic.go:356] "Generic (PLEG): container finished" podID="9150110d-a61e-4508-876d-bc3d0f8263c0" containerID="4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf" exitCode=0 Nov 02 20:21:54.363480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.363463 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" event={"ID":"9150110d-a61e-4508-876d-bc3d0f8263c0","Type":"ContainerDied","Data":"4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf"} Nov 02 20:21:54.363818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.363531 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" event={"ID":"9150110d-a61e-4508-876d-bc3d0f8263c0","Type":"ContainerDied","Data":"77d62876b25e71b9fa7bc52e30f23db2e5b4800c62c30c92527b8c040cbd35bd"} Nov 02 20:21:54.363818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.363561 5800 scope.go:122] "RemoveContainer" containerID="4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf" Nov 02 20:21:54.363818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.363769 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp" Nov 02 20:21:54.380193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.380173 5800 scope.go:122] "RemoveContainer" containerID="4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf" Nov 02 20:21:54.380500 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:21:54.380480 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf\": container with ID starting with 4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf not found: ID does not exist" containerID="4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf" Nov 02 20:21:54.380560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.380510 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf"} err="failed to get container status \"4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf\": rpc error: code = NotFound desc = could not find container \"4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf\": container with ID starting with 4061b70772d244d082b97dd638ff24c0b536c997256ced84a7db25cf7a188cbf not found: ID does not exist" Nov 02 20:21:54.517535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.517503 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp"] Nov 02 20:21:54.520529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:54.520500 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-777876bcb9-lcckp"] Nov 02 20:21:55.622662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:21:55.622624 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9150110d-a61e-4508-876d-bc3d0f8263c0" path="/var/lib/kubelet/pods/9150110d-a61e-4508-876d-bc3d0f8263c0/volumes" Nov 02 20:22:00.743020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.742986 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g"] Nov 02 20:22:00.744409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744378 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" containerName="payload-pre-processing" Nov 02 20:22:00.744409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744409 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" containerName="payload-pre-processing" Nov 02 20:22:00.744614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744463 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4c9f1d7c-50c7-49c2-b909-0e76b04414a3" containerName="revoke-keys" Nov 02 20:22:00.744614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744472 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9f1d7c-50c7-49c2-b909-0e76b04414a3" containerName="revoke-keys" Nov 02 20:22:00.744614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744570 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9150110d-a61e-4508-876d-bc3d0f8263c0" containerName="payload-processing" Nov 02 20:22:00.744614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744578 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9150110d-a61e-4508-876d-bc3d0f8263c0" containerName="payload-processing" Nov 02 20:22:00.744614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744610 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" containerName="payload-processing" Nov 02 20:22:00.744928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744618 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" containerName="payload-processing" Nov 02 20:22:00.744985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744943 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9150110d-a61e-4508-876d-bc3d0f8263c0" containerName="payload-processing" Nov 02 20:22:00.744985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744969 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87eaebf-e2ac-4a15-a657-b9ed0a23e7a8" containerName="payload-processing" Nov 02 20:22:00.745084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.744988 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f776f90-da6a-4a8b-9213-4b9fd8cc4d0d" containerName="payload-pre-processing" Nov 02 20:22:00.745084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.745001 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9f1d7c-50c7-49c2-b909-0e76b04414a3" containerName="revoke-keys" Nov 02 20:22:00.756593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.756563 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.805980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.805939 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-57e2cf-openshift-default-dockercfg-9df6f\"" Nov 02 20:22:00.855406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.855375 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g"] Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892193 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxcb\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-kube-api-access-mbxcb\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892320 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-data\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892374 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-socket\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892414 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-token\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892531 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-podinfo\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892596 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-credential-socket\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892642 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1173c959-8c68-4379-9ade-818f7d7e8fee-istiod-ca-cert\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892774 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-envoy\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.894303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.892823 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-certs\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994446 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-credential-socket\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994496 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1173c959-8c68-4379-9ade-818f7d7e8fee-istiod-ca-cert\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994549 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-envoy\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994574 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-certs\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994636 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxcb\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-kube-api-access-mbxcb\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994662 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-data\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994695 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-socket\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994727 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-token\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.994786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.994785 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-podinfo\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.995186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.995060 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-certs\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.995186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.995110 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-data\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.995328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.995222 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-credential-socket\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.995328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.995315 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-socket\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.995448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.995383 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1173c959-8c68-4379-9ade-818f7d7e8fee-istiod-ca-cert\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.996826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.996804 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-envoy\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:00.997274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:00.997253 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-podinfo\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:01.049808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:01.049779 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-token\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:01.049808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:01.049803 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxcb\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-kube-api-access-mbxcb\") pod \"e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:01.071722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:01.071693 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:01.489915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:01.489889 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g"] Nov 02 20:22:01.490523 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:22:01.490496 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1173c959_8c68_4379_9ade_818f7d7e8fee.slice/crio-4a3041c4e8f907d5b64590c6ac93de35c45683f4d370e9da540d0c1e2aef557c WatchSource:0}: Error finding container 4a3041c4e8f907d5b64590c6ac93de35c45683f4d370e9da540d0c1e2aef557c: Status 404 returned error can't find the container with id 4a3041c4e8f907d5b64590c6ac93de35c45683f4d370e9da540d0c1e2aef557c Nov 02 20:22:01.492876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:01.492843 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:01.492976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:01.492907 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:01.492976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:01.492937 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:02.430431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:02.430395 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" event={"ID":"1173c959-8c68-4379-9ade-818f7d7e8fee","Type":"ContainerStarted","Data":"32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471"} Nov 02 20:22:02.430838 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:02.430454 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" event={"ID":"1173c959-8c68-4379-9ade-818f7d7e8fee","Type":"ContainerStarted","Data":"4a3041c4e8f907d5b64590c6ac93de35c45683f4d370e9da540d0c1e2aef557c"} Nov 02 20:22:02.633539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:02.633481 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" podStartSLOduration=2.633463758 podStartE2EDuration="2.633463758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:02.584645928 +0000 UTC m=+2021.101418512" watchObservedRunningTime="2026-11-02 20:22:02.633463758 +0000 UTC m=+2021.150236342" Nov 02 20:22:03.072169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:03.072130 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:03.078202 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:03.078175 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:03.437855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:03.437757 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:03.438987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:03.438962 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:22:09.595992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.595961 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw"] Nov 02 20:22:09.600354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.600336 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.609348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.609320 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzbf7\" (UniqueName: \"kubernetes.io/projected/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-kube-api-access-vzbf7\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.609479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.609393 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.661812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.661777 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw"] Nov 02 20:22:09.758303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.750665 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzbf7\" (UniqueName: \"kubernetes.io/projected/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-kube-api-access-vzbf7\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.758303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.750742 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.758736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.758712 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-57e2cf\"" Nov 02 20:22:09.759076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.759063 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-57e2cf-dockercfg-qmxvw\"" Nov 02 20:22:09.761884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.761860 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.794258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.794227 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5"] Nov 02 20:22:09.803433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.803400 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:09.812779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.807673 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzbf7\" (UniqueName: \"kubernetes.io/projected/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-kube-api-access-vzbf7\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.818713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.818668 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5"] Nov 02 20:22:09.853025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.852890 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab607139-190f-4848-a5bb-bb62876c9bde-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:09.853025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.852994 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452lz\" (UniqueName: \"kubernetes.io/projected/ab607139-190f-4848-a5bb-bb62876c9bde-kube-api-access-452lz\") pod \"payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:09.916107 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.916024 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:09.955953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.955892 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab607139-190f-4848-a5bb-bb62876c9bde-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:09.956126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.956031 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452lz\" (UniqueName: \"kubernetes.io/projected/ab607139-190f-4848-a5bb-bb62876c9bde-kube-api-access-452lz\") pod \"payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:09.959018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:09.958972 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab607139-190f-4848-a5bb-bb62876c9bde-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:10.010826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.010790 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-452lz\" (UniqueName: \"kubernetes.io/projected/ab607139-190f-4848-a5bb-bb62876c9bde-kube-api-access-452lz\") pod \"payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:10.055305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.053824 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m"] Nov 02 20:22:10.059617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.059591 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.116294 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.116179 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-408103-openshift-default-dockercfg-dvwdd\"" Nov 02 20:22:10.125037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.125006 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m"] Nov 02 20:22:10.149806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.149744 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:10.158758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.158725 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-socket\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.158853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.158773 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-data\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.158960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.158935 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-envoy\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.159009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.158982 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-token\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.159152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.159134 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-certs\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.159205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.159189 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-kube-api-access-zfktq\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.159307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.159270 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-credential-socket\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.159514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.159490 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-podinfo\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.159628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.159536 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istiod-ca-cert\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275425 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-certs\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275494 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-kube-api-access-zfktq\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275535 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-credential-socket\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275568 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-podinfo\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275596 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istiod-ca-cert\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275653 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-socket\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275673 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-data\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275734 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-envoy\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.275753 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-token\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.276353 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-certs\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.277106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.276740 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-credential-socket\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.278245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.277328 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-socket\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.278906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.278883 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-data\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.279820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.279731 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istiod-ca-cert\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.282443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.281371 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-podinfo\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.282443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.281457 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-envoy\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.303478 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:22:10.303448 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc328d2b_6c7d_4ed2_b195_fb4659d7acc5.slice/crio-6c4a1c09fec60844614f99cb22a57dc0221e0b7c497dd24fd4198c45027d3337 WatchSource:0}: Error finding container 6c4a1c09fec60844614f99cb22a57dc0221e0b7c497dd24fd4198c45027d3337: Status 404 returned error can't find the container with id 6c4a1c09fec60844614f99cb22a57dc0221e0b7c497dd24fd4198c45027d3337 Nov 02 20:22:10.304444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.304406 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw"] Nov 02 20:22:10.330503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.330476 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-token\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.330749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.330724 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-kube-api-access-zfktq\") pod \"e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.378546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.378523 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:10.531865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.526674 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" event={"ID":"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5","Type":"ContainerStarted","Data":"63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d"} Nov 02 20:22:10.531865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.526771 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" event={"ID":"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5","Type":"ContainerStarted","Data":"6c4a1c09fec60844614f99cb22a57dc0221e0b7c497dd24fd4198c45027d3337"} Nov 02 20:22:10.531865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.526991 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:10.558048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.558012 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5"] Nov 02 20:22:10.562159 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:22:10.561994 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab607139_190f_4848_a5bb_bb62876c9bde.slice/crio-05000e530dfde80c7e1736b146de40a3c24a9202eca2ccb0833a75221fa03083 WatchSource:0}: Error finding container 05000e530dfde80c7e1736b146de40a3c24a9202eca2ccb0833a75221fa03083: Status 404 returned error can't find the container with id 05000e530dfde80c7e1736b146de40a3c24a9202eca2ccb0833a75221fa03083 Nov 02 20:22:10.779650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.779589 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" podStartSLOduration=1.77957316 podStartE2EDuration="1.77957316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:10.63897807 +0000 UTC m=+2029.155750656" watchObservedRunningTime="2026-11-02 20:22:10.77957316 +0000 UTC m=+2029.296345784" Nov 02 20:22:10.782148 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.782119 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m"] Nov 02 20:22:10.784306 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:22:10.784257 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e78e4c5_6892_4e97_9f64_8e0c6da82a17.slice/crio-6f732686a5ebfeb8a124d5aa29d26d6cd2b4aeb0593064d699b02d1a0e9806c3 WatchSource:0}: Error finding container 6f732686a5ebfeb8a124d5aa29d26d6cd2b4aeb0593064d699b02d1a0e9806c3: Status 404 returned error can't find the container with id 6f732686a5ebfeb8a124d5aa29d26d6cd2b4aeb0593064d699b02d1a0e9806c3 Nov 02 20:22:10.787582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.787542 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:10.787691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.787635 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:10.787691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:10.787675 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:22:11.537917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:11.537878 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" event={"ID":"ab607139-190f-4848-a5bb-bb62876c9bde","Type":"ContainerStarted","Data":"5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8"} Nov 02 20:22:11.538217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:11.537944 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" event={"ID":"ab607139-190f-4848-a5bb-bb62876c9bde","Type":"ContainerStarted","Data":"05000e530dfde80c7e1736b146de40a3c24a9202eca2ccb0833a75221fa03083"} Nov 02 20:22:11.538217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:11.537979 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:11.540001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:11.539973 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" event={"ID":"7e78e4c5-6892-4e97-9f64-8e0c6da82a17","Type":"ContainerStarted","Data":"a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b"} Nov 02 20:22:11.540201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:11.540178 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" event={"ID":"7e78e4c5-6892-4e97-9f64-8e0c6da82a17","Type":"ContainerStarted","Data":"6f732686a5ebfeb8a124d5aa29d26d6cd2b4aeb0593064d699b02d1a0e9806c3"} Nov 02 20:22:11.682628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:11.682552 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" podStartSLOduration=2.682532321 podStartE2EDuration="2.682532321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:11.674664548 +0000 UTC m=+2030.191437133" watchObservedRunningTime="2026-11-02 20:22:11.682532321 +0000 UTC m=+2030.199304911" Nov 02 20:22:11.791334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:11.791222 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" podStartSLOduration=1.791208633 podStartE2EDuration="1.791208633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:11.786654429 +0000 UTC m=+2030.303427016" watchObservedRunningTime="2026-11-02 20:22:11.791208633 +0000 UTC m=+2030.307981218" Nov 02 20:22:12.379171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:12.379136 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:12.384736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:12.384710 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:12.548641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:12.548608 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:12.549462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:12.549437 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:22:21.543432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:21.543401 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:22:22.552123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:22.552089 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:22:22.699222 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:22:22.699186 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:22:27.190799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:27.190759 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj"] Nov 02 20:22:27.191428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:27.190958 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" containerID="cri-o://b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a" gracePeriod=30 Nov 02 20:22:32.487446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.487410 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:22:32.625448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625417 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-certs\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.625448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625456 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-data\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.625764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625496 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-envoy\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.625764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625687 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-socket\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.625764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625738 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4591daec-4b8a-4047-b2d7-1e1826365b7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4591daec-4b8a-4047-b2d7-1e1826365b7e-istiod-ca-cert\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.626026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625763 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-certs" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:22:32.626026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625781 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-credential-socket\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.626026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625781 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-data" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:22:32.626026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625845 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-token\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.626026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625878 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-podinfo\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.626026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625896 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-socket" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:22:32.626026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.625904 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-kube-api-access-zrmhs\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-kube-api-access-zrmhs\") pod \"4591daec-4b8a-4047-b2d7-1e1826365b7e\" (UID: \"4591daec-4b8a-4047-b2d7-1e1826365b7e\") " Nov 02 20:22:32.626560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.626344 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-credential-socket" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:22:32.626560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.626446 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.626560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.626467 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.626560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.626481 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.626560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.626494 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.626865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.626585 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4591daec-4b8a-4047-b2d7-1e1826365b7e-istiod-ca-cert" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:22:32.628467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.628426 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-podinfo" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:22:32.628621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.628562 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-envoy" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:22:32.628621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.628596 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-token" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:22:32.629086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.629060 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-kube-api-access-zrmhs" pod "4591daec-4b8a-4047-b2d7-1e1826365b7e" (UID: "4591daec-4b8a-4047-b2d7-1e1826365b7e"). InnerVolumeSpecName "kube-api-access-zrmhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:22:32.728414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.727627 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4591daec-4b8a-4047-b2d7-1e1826365b7e-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.728414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.727660 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.728414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.727677 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.728414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.727692 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmhs\" (UniqueName: \"kubernetes.io/projected/4591daec-4b8a-4047-b2d7-1e1826365b7e-kube-api-access-zrmhs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.728414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.727708 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4591daec-4b8a-4047-b2d7-1e1826365b7e-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:22:32.741728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.741695 5800 generic.go:356] "Generic (PLEG): container finished" podID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerID="b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a" exitCode=0 Nov 02 20:22:32.741921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.741814 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" Nov 02 20:22:32.741921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.741837 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" event={"ID":"4591daec-4b8a-4047-b2d7-1e1826365b7e","Type":"ContainerDied","Data":"b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a"} Nov 02 20:22:32.741921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.741879 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj" event={"ID":"4591daec-4b8a-4047-b2d7-1e1826365b7e","Type":"ContainerDied","Data":"4641ed8669d180b6db2dfb194a60e59e539de9bde93133de417e06a197e85a0a"} Nov 02 20:22:32.741921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.741900 5800 scope.go:122] "RemoveContainer" containerID="b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a" Nov 02 20:22:32.754993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.754974 5800 scope.go:122] "RemoveContainer" containerID="b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a" Nov 02 20:22:32.755256 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:22:32.755230 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a\": container with ID starting with b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a not found: ID does not exist" containerID="b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a" Nov 02 20:22:32.755386 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.755266 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a"} err="failed to get container status \"b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a\": rpc error: code = NotFound desc = could not find container \"b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a\": container with ID starting with b0737d5cc91c6e4be4695c5ae66e22c796be27651ac2585cc2207b986e9e2a1a not found: ID does not exist" Nov 02 20:22:32.897842 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.897681 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj"] Nov 02 20:22:32.900860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:32.900839 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-gxxbj"] Nov 02 20:22:33.625007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:33.624972 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" path="/var/lib/kubelet/pods/4591daec-4b8a-4047-b2d7-1e1826365b7e/volumes" Nov 02 20:22:52.695174 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:22:52.695140 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:22:59.300768 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.300732 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x"] Nov 02 20:22:59.301507 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.301489 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" Nov 02 20:22:59.301566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.301509 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" Nov 02 20:22:59.301643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.301633 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4591daec-4b8a-4047-b2d7-1e1826365b7e" containerName="istio-proxy" Nov 02 20:22:59.305418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.305400 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.383747 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.383697 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg89v\" (UniqueName: \"kubernetes.io/projected/15b6acc9-1a58-4227-83ab-e21af65c8a53-kube-api-access-fg89v\") pod \"payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.383945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.383907 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b6acc9-1a58-4227-83ab-e21af65c8a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.401025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.400980 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-316e97\"" Nov 02 20:22:59.401025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.401010 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-316e97-dockercfg-5kw2p\"" Nov 02 20:22:59.404527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.404498 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x"] Nov 02 20:22:59.485041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.485003 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b6acc9-1a58-4227-83ab-e21af65c8a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.485228 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.485199 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg89v\" (UniqueName: \"kubernetes.io/projected/15b6acc9-1a58-4227-83ab-e21af65c8a53-kube-api-access-fg89v\") pod \"payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.485824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.485800 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b6acc9-1a58-4227-83ab-e21af65c8a53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.540931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.540893 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg89v\" (UniqueName: \"kubernetes.io/projected/15b6acc9-1a58-4227-83ab-e21af65c8a53-kube-api-access-fg89v\") pod \"payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.620155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.620075 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:22:59.972164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.972052 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h"] Nov 02 20:22:59.976585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.976563 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:22:59.989465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:22:59.989431 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x"] Nov 02 20:22:59.991152 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:22:59.991090 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b6acc9_1a58_4227_83ab_e21af65c8a53.slice/crio-74aa2e91ed838262923d4c5c5c4de38c75754db9f3ad0163ac05f4caf87b8ff8 WatchSource:0}: Error finding container 74aa2e91ed838262923d4c5c5c4de38c75754db9f3ad0163ac05f4caf87b8ff8: Status 404 returned error can't find the container with id 74aa2e91ed838262923d4c5c5c4de38c75754db9f3ad0163ac05f4caf87b8ff8 Nov 02 20:23:00.029664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.029631 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h"] Nov 02 20:23:00.101378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.101330 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.101552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.101503 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqj2z\" (UniqueName: \"kubernetes.io/projected/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-kube-api-access-cqj2z\") pod \"payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.203327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.203263 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqj2z\" (UniqueName: \"kubernetes.io/projected/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-kube-api-access-cqj2z\") pod \"payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.203610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.203509 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.204256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.204235 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.255764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.255730 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqj2z\" (UniqueName: \"kubernetes.io/projected/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-kube-api-access-cqj2z\") pod \"payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.292802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.292769 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.667090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.667060 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h"] Nov 02 20:23:00.668197 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:23:00.668168 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21aef7b_c2bf_4bdc_94ce_cd35995a55f2.slice/crio-6d758047459412b917ec464f0de6dccc8b307681fe0396a5e85639d7e4d264e7 WatchSource:0}: Error finding container 6d758047459412b917ec464f0de6dccc8b307681fe0396a5e85639d7e4d264e7: Status 404 returned error can't find the container with id 6d758047459412b917ec464f0de6dccc8b307681fe0396a5e85639d7e4d264e7 Nov 02 20:23:00.948123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.948011 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" event={"ID":"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2","Type":"ContainerStarted","Data":"17a84eed602bc8bce176169dc205a4ca95adbde0cb5504968734e56624b93122"} Nov 02 20:23:00.948330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.948246 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:00.948330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.948319 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" event={"ID":"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2","Type":"ContainerStarted","Data":"6d758047459412b917ec464f0de6dccc8b307681fe0396a5e85639d7e4d264e7"} Nov 02 20:23:00.949827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.949801 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" event={"ID":"15b6acc9-1a58-4227-83ab-e21af65c8a53","Type":"ContainerStarted","Data":"0df74e143d3bfa680732f6b9c7010eeeedd422efd67a10ee14a21f214df4aca5"} Nov 02 20:23:00.949945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.949847 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" event={"ID":"15b6acc9-1a58-4227-83ab-e21af65c8a53","Type":"ContainerStarted","Data":"74aa2e91ed838262923d4c5c5c4de38c75754db9f3ad0163ac05f4caf87b8ff8"} Nov 02 20:23:00.950003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:00.949954 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:23:01.056949 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:01.056889 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" podStartSLOduration=2.0568756009999998 podStartE2EDuration="2.056875601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:01.053992749 +0000 UTC m=+2079.570765337" watchObservedRunningTime="2026-11-02 20:23:01.056875601 +0000 UTC m=+2079.573648186" Nov 02 20:23:01.161810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:01.161757 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" podStartSLOduration=2.161740928 podStartE2EDuration="2.161740928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:01.158818041 +0000 UTC m=+2079.675590628" watchObservedRunningTime="2026-11-02 20:23:01.161740928 +0000 UTC m=+2079.678513546" Nov 02 20:23:11.958999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:11.958964 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:23:11.959499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:11.959155 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:23:22.009327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:22.009294 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:23:22.014330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:22.014302 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:23:22.696753 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:23:22.696718 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:23:22.987210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:22.987133 5800 scope.go:122] "RemoveContainer" containerID="eb96b29d54079a05650fb1c701b21a2a1bb7b3bc185db6e6dc4d7050654ee1b0" Nov 02 20:23:29.824359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:29.824327 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr"] Nov 02 20:23:30.160601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.160515 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" podUID="6b6c9cce-ee3f-4650-842c-d800b7051c33" containerName="payload-processing" containerID="cri-o://d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7" gracePeriod=30 Nov 02 20:23:30.575673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.575648 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:23:30.601006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.600975 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6c9cce-ee3f-4650-842c-d800b7051c33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6c9cce-ee3f-4650-842c-d800b7051c33-plugins-config-volume\") pod \"6b6c9cce-ee3f-4650-842c-d800b7051c33\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " Nov 02 20:23:30.601151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.601032 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6c9cce-ee3f-4650-842c-d800b7051c33-kube-api-access-dwqrc\" (UniqueName: \"kubernetes.io/projected/6b6c9cce-ee3f-4650-842c-d800b7051c33-kube-api-access-dwqrc\") pod \"6b6c9cce-ee3f-4650-842c-d800b7051c33\" (UID: \"6b6c9cce-ee3f-4650-842c-d800b7051c33\") " Nov 02 20:23:30.601396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.601349 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6c9cce-ee3f-4650-842c-d800b7051c33-plugins-config-volume" pod "6b6c9cce-ee3f-4650-842c-d800b7051c33" (UID: "6b6c9cce-ee3f-4650-842c-d800b7051c33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:30.601546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.601532 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6c9cce-ee3f-4650-842c-d800b7051c33-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:30.603259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.603233 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6c9cce-ee3f-4650-842c-d800b7051c33-kube-api-access-dwqrc" pod "6b6c9cce-ee3f-4650-842c-d800b7051c33" (UID: "6b6c9cce-ee3f-4650-842c-d800b7051c33"). InnerVolumeSpecName "kube-api-access-dwqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:30.703606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:30.703557 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqrc\" (UniqueName: \"kubernetes.io/projected/6b6c9cce-ee3f-4650-842c-d800b7051c33-kube-api-access-dwqrc\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:31.168897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.168858 5800 generic.go:356] "Generic (PLEG): container finished" podID="6b6c9cce-ee3f-4650-842c-d800b7051c33" containerID="d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7" exitCode=0 Nov 02 20:23:31.169325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.168924 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" Nov 02 20:23:31.169325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.168944 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" event={"ID":"6b6c9cce-ee3f-4650-842c-d800b7051c33","Type":"ContainerDied","Data":"d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7"} Nov 02 20:23:31.169325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.169037 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr" event={"ID":"6b6c9cce-ee3f-4650-842c-d800b7051c33","Type":"ContainerDied","Data":"00ba27ea618c13dada199266c37540bf6d9dd763b26410434c5481ba570a2cb3"} Nov 02 20:23:31.169325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.169065 5800 scope.go:122] "RemoveContainer" containerID="d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7" Nov 02 20:23:31.180688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.180667 5800 scope.go:122] "RemoveContainer" containerID="d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7" Nov 02 20:23:31.180984 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:23:31.180964 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7\": container with ID starting with d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7 not found: ID does not exist" containerID="d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7" Nov 02 20:23:31.181029 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.180993 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7"} err="failed to get container status \"d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7\": rpc error: code = NotFound desc = could not find container \"d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7\": container with ID starting with d6f326b33e1548e731b95b79711986faf6d85d126cb5d68181baa2cf6d2d48a7 not found: ID does not exist" Nov 02 20:23:31.323144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.323098 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr"] Nov 02 20:23:31.369533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.369488 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a999d7-59b876789b-kttvr"] Nov 02 20:23:31.643479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:31.643444 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6c9cce-ee3f-4650-842c-d800b7051c33" path="/var/lib/kubelet/pods/6b6c9cce-ee3f-4650-842c-d800b7051c33/volumes" Nov 02 20:23:40.045801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:40.045762 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm"] Nov 02 20:23:40.089894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:40.089859 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf06014-24wqm"] Nov 02 20:23:40.519821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:40.519766 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22"] Nov 02 20:23:40.520061 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:40.520038 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" containerID="cri-o://33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6" gracePeriod=30 Nov 02 20:23:41.523087 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:41.523048 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:23:41.523590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:41.523120 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:23:41.622960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:41.622925 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463c4552-57cd-49d6-b1f2-b2f80621b910" path="/var/lib/kubelet/pods/463c4552-57cd-49d6-b1f2-b2f80621b910/volumes" Nov 02 20:23:45.914964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:45.914932 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:23:46.056783 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.056747 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-data\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.056986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.056797 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-token\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.056986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.056861 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-socket\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.057153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057001 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istiod-ca-cert\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.057153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057059 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-certs\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.057153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057089 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-credential-socket\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.057153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057015 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-data" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:46.057435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057169 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-socket" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:46.057435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057223 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-podinfo\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.057435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057269 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istiod-ca-cert" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:46.057435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057309 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-certs" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:46.057435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057341 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-envoy\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.057435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057406 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-credential-socket" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:46.057745 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057465 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-kube-api-access-zwstv\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-kube-api-access-zwstv\") pod \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\" (UID: \"e5ca928f-ece7-4c63-aae8-3aa97f7becd7\") " Nov 02 20:23:46.057972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057936 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.057972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057960 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.058130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057977 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.058130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.057991 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.058130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.058003 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.059741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.059713 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-token" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:46.059941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.059917 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-kube-api-access-zwstv" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "kube-api-access-zwstv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:46.059941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.059921 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-envoy" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:23:46.060674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.060656 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-podinfo" pod "e5ca928f-ece7-4c63-aae8-3aa97f7becd7" (UID: "e5ca928f-ece7-4c63-aae8-3aa97f7becd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:23:46.159820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.159733 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.159820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.159768 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.159820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.159777 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.159820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.159789 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwstv\" (UniqueName: \"kubernetes.io/projected/e5ca928f-ece7-4c63-aae8-3aa97f7becd7-kube-api-access-zwstv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:46.280720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.280680 5800 generic.go:356] "Generic (PLEG): container finished" podID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerID="33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6" exitCode=0 Nov 02 20:23:46.280911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.280728 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" Nov 02 20:23:46.280911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.280761 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" event={"ID":"e5ca928f-ece7-4c63-aae8-3aa97f7becd7","Type":"ContainerDied","Data":"33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6"} Nov 02 20:23:46.280911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.280815 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22" event={"ID":"e5ca928f-ece7-4c63-aae8-3aa97f7becd7","Type":"ContainerDied","Data":"064163519a3ef9417a93632cdced1eda05cb5f6ca12f37709fb1e0ca4314f514"} Nov 02 20:23:46.280911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.280836 5800 scope.go:122] "RemoveContainer" containerID="33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6" Nov 02 20:23:46.292104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.292087 5800 scope.go:122] "RemoveContainer" containerID="33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6" Nov 02 20:23:46.292409 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:23:46.292392 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6\": container with ID starting with 33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6 not found: ID does not exist" containerID="33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6" Nov 02 20:23:46.292473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.292417 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6"} err="failed to get container status \"33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6\": rpc error: code = NotFound desc = could not find container \"33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6\": container with ID starting with 33662c11abbef326763aa3153b31114164880650ac9eb219fbb4a02ab6b6a0e6 not found: ID does not exist" Nov 02 20:23:46.461698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.461588 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22"] Nov 02 20:23:46.496961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:46.496927 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-kgd22"] Nov 02 20:23:47.624755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:47.624722 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" path="/var/lib/kubelet/pods/e5ca928f-ece7-4c63-aae8-3aa97f7becd7/volumes" Nov 02 20:23:52.702469 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:23:52.702434 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:23:53.337396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.337301 5800 generic.go:356] "Generic (PLEG): container finished" podID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerID="32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471" exitCode=0 Nov 02 20:23:53.337396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.337373 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" event={"ID":"1173c959-8c68-4379-9ade-818f7d7e8fee","Type":"ContainerDied","Data":"32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471"} Nov 02 20:23:53.337878 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.337860 5800 scope.go:122] "RemoveContainer" containerID="32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471" Nov 02 20:23:53.339791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.339756 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:23:53.339900 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.339845 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:23:53.339900 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.339893 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:23:53.798068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.798038 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8"] Nov 02 20:23:53.798689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.798672 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6b6c9cce-ee3f-4650-842c-d800b7051c33" containerName="payload-processing" Nov 02 20:23:53.798689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.798690 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6c9cce-ee3f-4650-842c-d800b7051c33" containerName="payload-processing" Nov 02 20:23:53.798803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.798717 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" Nov 02 20:23:53.798803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.798723 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" Nov 02 20:23:53.798869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.798819 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6c9cce-ee3f-4650-842c-d800b7051c33" containerName="payload-processing" Nov 02 20:23:53.798869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.798831 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ca928f-ece7-4c63-aae8-3aa97f7becd7" containerName="istio-proxy" Nov 02 20:23:53.802136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.802116 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:53.851968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.851275 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3b8db2\"" Nov 02 20:23:53.852948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.852919 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3b8db2-dockercfg-psss4\"" Nov 02 20:23:53.856232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.856206 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8"] Nov 02 20:23:53.882846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.882808 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67z6d\" (UniqueName: \"kubernetes.io/projected/86d7073d-22ad-44d5-b02d-4c5e27728c40-kube-api-access-67z6d\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:53.883053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.883030 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7073d-22ad-44d5-b02d-4c5e27728c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:53.983820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.983782 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7073d-22ad-44d5-b02d-4c5e27728c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:53.983999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.983865 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67z6d\" (UniqueName: \"kubernetes.io/projected/86d7073d-22ad-44d5-b02d-4c5e27728c40-kube-api-access-67z6d\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:53.984630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:53.984604 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7073d-22ad-44d5-b02d-4c5e27728c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:54.036889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.036855 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67z6d\" (UniqueName: \"kubernetes.io/projected/86d7073d-22ad-44d5-b02d-4c5e27728c40-kube-api-access-67z6d\") pod \"payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:54.116306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.116196 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:54.356331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.355494 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" event={"ID":"1173c959-8c68-4379-9ade-818f7d7e8fee","Type":"ContainerStarted","Data":"5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd"} Nov 02 20:23:54.547305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.527507 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8"] Nov 02 20:23:54.573129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.572618 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2"] Nov 02 20:23:54.580925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.580904 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:54.646632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.646545 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2"] Nov 02 20:23:54.695894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.695391 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4472f-43aa-44b6-b2c0-8f57814488e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:54.695894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.695535 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprpj\" (UniqueName: \"kubernetes.io/projected/fad4472f-43aa-44b6-b2c0-8f57814488e9-kube-api-access-nprpj\") pod \"payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:54.803369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.802085 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4472f-43aa-44b6-b2c0-8f57814488e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:54.803369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.802236 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprpj\" (UniqueName: \"kubernetes.io/projected/fad4472f-43aa-44b6-b2c0-8f57814488e9-kube-api-access-nprpj\") pod \"payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:54.803954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.803528 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4472f-43aa-44b6-b2c0-8f57814488e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:54.858921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.858884 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprpj\" (UniqueName: \"kubernetes.io/projected/fad4472f-43aa-44b6-b2c0-8f57814488e9-kube-api-access-nprpj\") pod \"payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:54.907631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:54.907413 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:55.073757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.073670 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:23:55.073933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.073791 5800 patch_prober.go:29] interesting pod/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:23:55.073933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.073835 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 20:23:55.283138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.283103 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh"] Nov 02 20:23:55.283351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.283328 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" podUID="f09d0560-6c7d-47ce-8015-ca0446aff452" containerName="payload-pre-processing" containerID="cri-o://ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362" gracePeriod=30 Nov 02 20:23:55.383365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.383221 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" event={"ID":"86d7073d-22ad-44d5-b02d-4c5e27728c40","Type":"ContainerStarted","Data":"ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba"} Nov 02 20:23:55.383823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.383794 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" event={"ID":"86d7073d-22ad-44d5-b02d-4c5e27728c40","Type":"ContainerStarted","Data":"bceda30a278c2aa65b851d163a337564e286bdf0d76119e36c685b73049dccf2"} Nov 02 20:23:55.383928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.383882 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:23:55.414200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.414034 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2"] Nov 02 20:23:55.571259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.563636 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" podStartSLOduration=2.563620932 podStartE2EDuration="2.563620932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:55.561244538 +0000 UTC m=+2134.078017124" watchObservedRunningTime="2026-11-02 20:23:55.563620932 +0000 UTC m=+2134.080393517" Nov 02 20:23:55.779432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.779383 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:23:55.869308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.868520 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09d0560-6c7d-47ce-8015-ca0446aff452-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09d0560-6c7d-47ce-8015-ca0446aff452-plugins-config-volume\") pod \"f09d0560-6c7d-47ce-8015-ca0446aff452\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " Nov 02 20:23:55.869308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.868637 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09d0560-6c7d-47ce-8015-ca0446aff452-kube-api-access-pp7h6\" (UniqueName: \"kubernetes.io/projected/f09d0560-6c7d-47ce-8015-ca0446aff452-kube-api-access-pp7h6\") pod \"f09d0560-6c7d-47ce-8015-ca0446aff452\" (UID: \"f09d0560-6c7d-47ce-8015-ca0446aff452\") " Nov 02 20:23:55.869887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.869813 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09d0560-6c7d-47ce-8015-ca0446aff452-plugins-config-volume" pod "f09d0560-6c7d-47ce-8015-ca0446aff452" (UID: "f09d0560-6c7d-47ce-8015-ca0446aff452"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:55.871701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.871669 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09d0560-6c7d-47ce-8015-ca0446aff452-kube-api-access-pp7h6" pod "f09d0560-6c7d-47ce-8015-ca0446aff452" (UID: "f09d0560-6c7d-47ce-8015-ca0446aff452"). InnerVolumeSpecName "kube-api-access-pp7h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:55.970931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.970894 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7h6\" (UniqueName: \"kubernetes.io/projected/f09d0560-6c7d-47ce-8015-ca0446aff452-kube-api-access-pp7h6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:55.971186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:55.971168 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09d0560-6c7d-47ce-8015-ca0446aff452-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:56.090670 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.072988 5800 patch_prober.go:29] interesting pod/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:23:56.090670 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.073047 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 20:23:56.312498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.312460 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j"] Nov 02 20:23:56.312780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.312743 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" containerID="cri-o://2fc0fbd7acf0a816f780999ace61bbde08905aa57859912a74b2ce81a3169e95" gracePeriod=30 Nov 02 20:23:56.394018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.393922 5800 generic.go:356] "Generic (PLEG): container finished" podID="f09d0560-6c7d-47ce-8015-ca0446aff452" containerID="ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362" exitCode=0 Nov 02 20:23:56.394219 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.394195 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" event={"ID":"f09d0560-6c7d-47ce-8015-ca0446aff452","Type":"ContainerDied","Data":"ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362"} Nov 02 20:23:56.394318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.394256 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" event={"ID":"f09d0560-6c7d-47ce-8015-ca0446aff452","Type":"ContainerDied","Data":"4a8825996057c6181764a40a4929ae07ec21b800ec4a562922b4643d038dfeb8"} Nov 02 20:23:56.394318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.394303 5800 scope.go:122] "RemoveContainer" containerID="ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362" Nov 02 20:23:56.394500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.394482 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh" Nov 02 20:23:56.398626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.398593 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" event={"ID":"fad4472f-43aa-44b6-b2c0-8f57814488e9","Type":"ContainerStarted","Data":"736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e"} Nov 02 20:23:56.398733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.398643 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" event={"ID":"fad4472f-43aa-44b6-b2c0-8f57814488e9","Type":"ContainerStarted","Data":"25c6e1e5589922c235fe8d97de30d58da3c46e5d958ea0ab569ad0e24875c313"} Nov 02 20:23:56.398846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.398832 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:23:56.408187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.408149 5800 scope.go:122] "RemoveContainer" containerID="ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362" Nov 02 20:23:56.408461 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:23:56.408440 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362\": container with ID starting with ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362 not found: ID does not exist" containerID="ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362" Nov 02 20:23:56.408509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.408472 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362"} err="failed to get container status \"ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362\": rpc error: code = NotFound desc = could not find container \"ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362\": container with ID starting with ade8a24fe72d2c272d839960c70c31f061bbcb440310a2cec1b50c23b9efe362 not found: ID does not exist" Nov 02 20:23:56.584303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.571500 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" podStartSLOduration=2.571478301 podStartE2EDuration="2.571478301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:56.562041023 +0000 UTC m=+2135.078813608" watchObservedRunningTime="2026-11-02 20:23:56.571478301 +0000 UTC m=+2135.088250889" Nov 02 20:23:56.688808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.688628 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh"] Nov 02 20:23:56.693313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:56.693249 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-296ae9-77859bdf55-s4xvh"] Nov 02 20:23:57.073086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:57.073047 5800 patch_prober.go:29] interesting pod/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:23:57.073560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:57.073132 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 20:23:57.645308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:57.625864 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09d0560-6c7d-47ce-8015-ca0446aff452" path="/var/lib/kubelet/pods/f09d0560-6c7d-47ce-8015-ca0446aff452/volumes" Nov 02 20:23:58.073118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:58.073076 5800 patch_prober.go:29] interesting pod/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:23:58.073632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:58.073163 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 20:23:59.072601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:59.072558 5800 patch_prober.go:29] interesting pod/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 02 20:23:59.072925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:23:59.072886 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 02 20:24:00.077265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:00.077232 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:24:00.077709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:00.077651 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:24:00.078491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:00.078468 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:24:01.488217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:01.488182 5800 generic.go:356] "Generic (PLEG): container finished" podID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerID="2fc0fbd7acf0a816f780999ace61bbde08905aa57859912a74b2ce81a3169e95" exitCode=0 Nov 02 20:24:01.488669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:01.488252 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" event={"ID":"e60d3f55-ff2f-4876-8a0a-3a88340cefe1","Type":"ContainerDied","Data":"2fc0fbd7acf0a816f780999ace61bbde08905aa57859912a74b2ce81a3169e95"} Nov 02 20:24:01.488669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:01.488370 5800 scope.go:122] "RemoveContainer" containerID="1d1f1d11b9c1a7b4dea4602eac0a8495fd7a077c9a8da1217d5608beb5d3e49c" Nov 02 20:24:02.000173 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.000148 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:24:02.105776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.105685 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-socket\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.105969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.105776 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-credential-socket\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.105969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.105827 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-kube-api-access-znbcm\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-kube-api-access-znbcm\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.105969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.105878 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-certs\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.106171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.105962 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-socket" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:02.106171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.105971 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istiod-ca-cert\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.106171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106123 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-credential-socket" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:02.106171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106149 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-data\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.106392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106212 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-certs" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:02.106392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106250 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-envoy\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.106392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106320 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-token\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-token\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.106392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106369 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-podinfo\") pod \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\" (UID: \"e60d3f55-ff2f-4876-8a0a-3a88340cefe1\") " Nov 02 20:24:02.106581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106416 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-data" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:02.106581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106356 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istiod-ca-cert" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:02.106922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106901 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.106922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106922 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.107096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106934 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.107096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106948 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.107096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.106959 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.108548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.108521 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-token" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:02.108757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.108734 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-kube-api-access-znbcm" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "kube-api-access-znbcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:02.108875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.108855 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-podinfo" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:24:02.108936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.108894 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-envoy" pod "e60d3f55-ff2f-4876-8a0a-3a88340cefe1" (UID: "e60d3f55-ff2f-4876-8a0a-3a88340cefe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:02.208056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.208011 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.208056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.208056 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.208379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.208073 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znbcm\" (UniqueName: \"kubernetes.io/projected/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-kube-api-access-znbcm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.208379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.208090 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e60d3f55-ff2f-4876-8a0a-3a88340cefe1-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:02.497436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.497345 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" Nov 02 20:24:02.497436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.497380 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j" event={"ID":"e60d3f55-ff2f-4876-8a0a-3a88340cefe1","Type":"ContainerDied","Data":"0f371b0e2dacfef95b09c3df9fc93e8ddd9b5d9880deaba4b3e57994f20a6ba7"} Nov 02 20:24:02.497977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.497450 5800 scope.go:122] "RemoveContainer" containerID="2fc0fbd7acf0a816f780999ace61bbde08905aa57859912a74b2ce81a3169e95" Nov 02 20:24:02.656679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.656632 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j"] Nov 02 20:24:02.703090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:02.703061 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d3184c-openshift-default-67f95d8f7f-hsj6j"] Nov 02 20:24:03.622782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:03.622745 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" path="/var/lib/kubelet/pods/e60d3f55-ff2f-4876-8a0a-3a88340cefe1/volumes" Nov 02 20:24:06.402336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:06.402303 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:24:07.268841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.268806 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h"] Nov 02 20:24:07.269105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.269082 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" podUID="f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" containerName="payload-processing" containerID="cri-o://17a84eed602bc8bce176169dc205a4ca95adbde0cb5504968734e56624b93122" gracePeriod=30 Nov 02 20:24:07.367528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.367489 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x"] Nov 02 20:24:07.367713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.367693 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" podUID="15b6acc9-1a58-4227-83ab-e21af65c8a53" containerName="payload-pre-processing" containerID="cri-o://0df74e143d3bfa680732f6b9c7010eeeedd422efd67a10ee14a21f214df4aca5" gracePeriod=30 Nov 02 20:24:07.409490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.409460 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:24:07.553394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.553267 5800 generic.go:356] "Generic (PLEG): container finished" podID="15b6acc9-1a58-4227-83ab-e21af65c8a53" containerID="0df74e143d3bfa680732f6b9c7010eeeedd422efd67a10ee14a21f214df4aca5" exitCode=0 Nov 02 20:24:07.553566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.553315 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" event={"ID":"15b6acc9-1a58-4227-83ab-e21af65c8a53","Type":"ContainerDied","Data":"0df74e143d3bfa680732f6b9c7010eeeedd422efd67a10ee14a21f214df4aca5"} Nov 02 20:24:07.555784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.555759 5800 generic.go:356] "Generic (PLEG): container finished" podID="f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" containerID="17a84eed602bc8bce176169dc205a4ca95adbde0cb5504968734e56624b93122" exitCode=0 Nov 02 20:24:07.556102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.555975 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" event={"ID":"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2","Type":"ContainerDied","Data":"17a84eed602bc8bce176169dc205a4ca95adbde0cb5504968734e56624b93122"} Nov 02 20:24:07.620339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.619332 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:24:07.683087 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.683060 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:24:07.694887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.694861 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-kube-api-access-cqj2z\" (UniqueName: \"kubernetes.io/projected/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-kube-api-access-cqj2z\") pod \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " Nov 02 20:24:07.695049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.694916 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-plugins-config-volume\") pod \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\" (UID: \"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2\") " Nov 02 20:24:07.695049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.694935 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b6acc9-1a58-4227-83ab-e21af65c8a53-kube-api-access-fg89v\" (UniqueName: \"kubernetes.io/projected/15b6acc9-1a58-4227-83ab-e21af65c8a53-kube-api-access-fg89v\") pod \"15b6acc9-1a58-4227-83ab-e21af65c8a53\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " Nov 02 20:24:07.695049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.694996 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b6acc9-1a58-4227-83ab-e21af65c8a53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b6acc9-1a58-4227-83ab-e21af65c8a53-plugins-config-volume\") pod \"15b6acc9-1a58-4227-83ab-e21af65c8a53\" (UID: \"15b6acc9-1a58-4227-83ab-e21af65c8a53\") " Nov 02 20:24:07.695478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.695444 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-plugins-config-volume" pod "f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" (UID: "f21aef7b-c2bf-4bdc-94ce-cd35995a55f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:07.695609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.695495 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b6acc9-1a58-4227-83ab-e21af65c8a53-plugins-config-volume" pod "15b6acc9-1a58-4227-83ab-e21af65c8a53" (UID: "15b6acc9-1a58-4227-83ab-e21af65c8a53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:07.695706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.695681 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:07.695766 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.695713 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15b6acc9-1a58-4227-83ab-e21af65c8a53-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:07.697251 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.697218 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b6acc9-1a58-4227-83ab-e21af65c8a53-kube-api-access-fg89v" pod "15b6acc9-1a58-4227-83ab-e21af65c8a53" (UID: "15b6acc9-1a58-4227-83ab-e21af65c8a53"). InnerVolumeSpecName "kube-api-access-fg89v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:07.697374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.697262 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-kube-api-access-cqj2z" pod "f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" (UID: "f21aef7b-c2bf-4bdc-94ce-cd35995a55f2"). InnerVolumeSpecName "kube-api-access-cqj2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:07.799035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.798994 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqj2z\" (UniqueName: \"kubernetes.io/projected/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2-kube-api-access-cqj2z\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:07.799035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:07.799037 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg89v\" (UniqueName: \"kubernetes.io/projected/15b6acc9-1a58-4227-83ab-e21af65c8a53-kube-api-access-fg89v\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:08.567903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.567805 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" Nov 02 20:24:08.567903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.567846 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x" event={"ID":"15b6acc9-1a58-4227-83ab-e21af65c8a53","Type":"ContainerDied","Data":"74aa2e91ed838262923d4c5c5c4de38c75754db9f3ad0163ac05f4caf87b8ff8"} Nov 02 20:24:08.568692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.567922 5800 scope.go:122] "RemoveContainer" containerID="0df74e143d3bfa680732f6b9c7010eeeedd422efd67a10ee14a21f214df4aca5" Nov 02 20:24:08.573600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.573575 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" Nov 02 20:24:08.573736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.573657 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h" event={"ID":"f21aef7b-c2bf-4bdc-94ce-cd35995a55f2","Type":"ContainerDied","Data":"6d758047459412b917ec464f0de6dccc8b307681fe0396a5e85639d7e4d264e7"} Nov 02 20:24:08.580946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.580927 5800 scope.go:122] "RemoveContainer" containerID="17a84eed602bc8bce176169dc205a4ca95adbde0cb5504968734e56624b93122" Nov 02 20:24:08.722330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.722298 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x"] Nov 02 20:24:08.768541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.768510 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-316e97-b5478d4b7-v9x9x"] Nov 02 20:24:08.872366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.871211 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h"] Nov 02 20:24:08.874703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:08.874675 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-316e97-6b8d7f6dbb-rdr7h"] Nov 02 20:24:09.630982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:09.630941 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b6acc9-1a58-4227-83ab-e21af65c8a53" path="/var/lib/kubelet/pods/15b6acc9-1a58-4227-83ab-e21af65c8a53/volumes" Nov 02 20:24:09.631570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:09.631549 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" path="/var/lib/kubelet/pods/f21aef7b-c2bf-4bdc-94ce-cd35995a55f2/volumes" Nov 02 20:24:13.205437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:13.205401 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk"] Nov 02 20:24:13.205866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:13.205626 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" containerID="cri-o://7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1" gracePeriod=30 Nov 02 20:24:18.490270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.490197 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:24:18.640725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.640693 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-credential-socket\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.640939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.640743 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-socket\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.640939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.640780 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-data\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.640939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.640801 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-kube-api-access-np6nm\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-kube-api-access-np6nm\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.640939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.640824 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-podinfo\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.640939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.640877 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-certs\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.640939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.640929 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8571aea-4c23-457c-a12c-4cc346a63e4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8571aea-4c23-457c-a12c-4cc346a63e4c-istiod-ca-cert\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.641371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641044 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-credential-socket" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:18.641371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641066 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-data" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:18.641371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641138 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-envoy\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.641371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641173 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-token\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-token\") pod \"d8571aea-4c23-457c-a12c-4cc346a63e4c\" (UID: \"d8571aea-4c23-457c-a12c-4cc346a63e4c\") " Nov 02 20:24:18.641371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641244 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-socket" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:18.641371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641308 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-certs" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:18.641695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641618 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8571aea-4c23-457c-a12c-4cc346a63e4c-istiod-ca-cert" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:18.641872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641846 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.641953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641878 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.641953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641893 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.641953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641907 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.641953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.641921 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8571aea-4c23-457c-a12c-4cc346a63e4c-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.643722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.643698 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-kube-api-access-np6nm" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "kube-api-access-np6nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:18.643888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.643865 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-token" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:18.643965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.643868 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-envoy" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:24:18.644015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.643962 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-podinfo" pod "d8571aea-4c23-457c-a12c-4cc346a63e4c" (UID: "d8571aea-4c23-457c-a12c-4cc346a63e4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:24:18.687236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.687198 5800 generic.go:356] "Generic (PLEG): container finished" podID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerID="7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1" exitCode=0 Nov 02 20:24:18.687393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.687253 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" Nov 02 20:24:18.687393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.687297 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" event={"ID":"d8571aea-4c23-457c-a12c-4cc346a63e4c","Type":"ContainerDied","Data":"7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1"} Nov 02 20:24:18.687393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.687357 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk" event={"ID":"d8571aea-4c23-457c-a12c-4cc346a63e4c","Type":"ContainerDied","Data":"6bc9438196f51e39b69fe07716bf238bf70dece7118a6174e9c8afef46f420f0"} Nov 02 20:24:18.687393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.687383 5800 scope.go:122] "RemoveContainer" containerID="7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1" Nov 02 20:24:18.701556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.701536 5800 scope.go:122] "RemoveContainer" containerID="33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160" Nov 02 20:24:18.711794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.711772 5800 scope.go:122] "RemoveContainer" containerID="7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1" Nov 02 20:24:18.712092 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:24:18.712064 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1\": container with ID starting with 7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1 not found: ID does not exist" containerID="7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1" Nov 02 20:24:18.712141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.712098 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1"} err="failed to get container status \"7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1\": rpc error: code = NotFound desc = could not find container \"7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1\": container with ID starting with 7b3955addbec31c55c5c5187c70ecf471fbbde9e5aa62bb7b9cafda2c8eb9cb1 not found: ID does not exist" Nov 02 20:24:18.712141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.712119 5800 scope.go:122] "RemoveContainer" containerID="33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160" Nov 02 20:24:18.712390 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:24:18.712372 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160\": container with ID starting with 33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160 not found: ID does not exist" containerID="33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160" Nov 02 20:24:18.712449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.712394 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160"} err="failed to get container status \"33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160\": rpc error: code = NotFound desc = could not find container \"33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160\": container with ID starting with 33daa74caf5a2e39527b00358342a2c41915aa4c882478c4beb3a30663e34160 not found: ID does not exist" Nov 02 20:24:18.743516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.743439 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np6nm\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-kube-api-access-np6nm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.743516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.743468 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.743516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.743484 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.743516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.743500 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8571aea-4c23-457c-a12c-4cc346a63e4c-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:18.841262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.841196 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk"] Nov 02 20:24:18.846445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:18.846409 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-316e97-openshift-default-58cfbb97f5-5zjzk"] Nov 02 20:24:19.623548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:19.623516 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" path="/var/lib/kubelet/pods/d8571aea-4c23-457c-a12c-4cc346a63e4c/volumes" Nov 02 20:24:22.696660 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:24:22.696623 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:24:23.715817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.714073 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z"] Nov 02 20:24:23.717535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.717507 5800 cpu_manager.go:414] "containerMap: removing container" podUID="15b6acc9-1a58-4227-83ab-e21af65c8a53" containerName="payload-pre-processing" Nov 02 20:24:23.717759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.717740 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b6acc9-1a58-4227-83ab-e21af65c8a53" containerName="payload-pre-processing" Nov 02 20:24:23.717877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.717866 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" containerName="payload-processing" Nov 02 20:24:23.717972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.717961 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" containerName="payload-processing" Nov 02 20:24:23.718144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718134 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" Nov 02 20:24:23.718235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718223 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" Nov 02 20:24:23.718347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718335 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" Nov 02 20:24:23.718453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718441 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" Nov 02 20:24:23.718556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718546 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" Nov 02 20:24:23.718641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718631 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" Nov 02 20:24:23.718733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718723 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" Nov 02 20:24:23.718816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718806 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" Nov 02 20:24:23.718913 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718904 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f09d0560-6c7d-47ce-8015-ca0446aff452" containerName="payload-pre-processing" Nov 02 20:24:23.719006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.718996 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09d0560-6c7d-47ce-8015-ca0446aff452" containerName="payload-pre-processing" Nov 02 20:24:23.719443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.719421 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" Nov 02 20:24:23.719547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.719538 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" Nov 02 20:24:23.719649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.719640 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8571aea-4c23-457c-a12c-4cc346a63e4c" containerName="istio-proxy" Nov 02 20:24:23.719743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.719733 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60d3f55-ff2f-4876-8a0a-3a88340cefe1" containerName="istio-proxy" Nov 02 20:24:23.719845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.719835 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09d0560-6c7d-47ce-8015-ca0446aff452" containerName="payload-pre-processing" Nov 02 20:24:23.719924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.719915 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b6acc9-1a58-4227-83ab-e21af65c8a53" containerName="payload-pre-processing" Nov 02 20:24:23.720016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.720005 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21aef7b-c2bf-4bdc-94ce-cd35995a55f2" containerName="payload-processing" Nov 02 20:24:23.726247 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.726221 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:23.774785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.774745 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z"] Nov 02 20:24:23.814262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.814065 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd42s\" (UniqueName: \"kubernetes.io/projected/38b428d7-ab0f-480d-bcfe-0779fb106abf-kube-api-access-dd42s\") pod \"payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:23.814262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.814111 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b028e6\"" Nov 02 20:24:23.814262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.814176 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b028e6-dockercfg-dz279\"" Nov 02 20:24:23.814262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.814234 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b428d7-ab0f-480d-bcfe-0779fb106abf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:23.861118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.861077 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h"] Nov 02 20:24:23.865452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.865431 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:23.915137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.915107 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b428d7-ab0f-480d-bcfe-0779fb106abf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:23.915448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.915195 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd42s\" (UniqueName: \"kubernetes.io/projected/38b428d7-ab0f-480d-bcfe-0779fb106abf-kube-api-access-dd42s\") pod \"payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:23.915901 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.915876 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b428d7-ab0f-480d-bcfe-0779fb106abf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:23.916432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.916408 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h"] Nov 02 20:24:23.968245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:23.968169 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd42s\" (UniqueName: \"kubernetes.io/projected/38b428d7-ab0f-480d-bcfe-0779fb106abf-kube-api-access-dd42s\") pod \"payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:24.016606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.016572 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944cd387-111f-4550-a4db-bae84d5fea95-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.016767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.016733 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw42p\" (UniqueName: \"kubernetes.io/projected/944cd387-111f-4550-a4db-bae84d5fea95-kube-api-access-xw42p\") pod \"payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.059776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.059744 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:24.118413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.118372 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw42p\" (UniqueName: \"kubernetes.io/projected/944cd387-111f-4550-a4db-bae84d5fea95-kube-api-access-xw42p\") pod \"payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.118604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.118586 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944cd387-111f-4550-a4db-bae84d5fea95-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.119564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.119538 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944cd387-111f-4550-a4db-bae84d5fea95-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.171522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.171488 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw42p\" (UniqueName: \"kubernetes.io/projected/944cd387-111f-4550-a4db-bae84d5fea95-kube-api-access-xw42p\") pod \"payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.180126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.180094 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.429131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.429098 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z"] Nov 02 20:24:24.429268 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:24:24.429155 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b428d7_ab0f_480d_bcfe_0779fb106abf.slice/crio-88065f9bfba95c7a22eb21eee7320a0466e8d7e4a40d621088e19659e5e7fef6 WatchSource:0}: Error finding container 88065f9bfba95c7a22eb21eee7320a0466e8d7e4a40d621088e19659e5e7fef6: Status 404 returned error can't find the container with id 88065f9bfba95c7a22eb21eee7320a0466e8d7e4a40d621088e19659e5e7fef6 Nov 02 20:24:24.549437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.549408 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h"] Nov 02 20:24:24.550793 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:24:24.550766 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944cd387_111f_4550_a4db_bae84d5fea95.slice/crio-b3f4b78aab3dfa297afd7baa2ff6247fc43d2d375f835515a968b86a822c1ff7 WatchSource:0}: Error finding container b3f4b78aab3dfa297afd7baa2ff6247fc43d2d375f835515a968b86a822c1ff7: Status 404 returned error can't find the container with id b3f4b78aab3dfa297afd7baa2ff6247fc43d2d375f835515a968b86a822c1ff7 Nov 02 20:24:24.749750 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.749718 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" event={"ID":"38b428d7-ab0f-480d-bcfe-0779fb106abf","Type":"ContainerStarted","Data":"a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae"} Nov 02 20:24:24.750230 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.749779 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" event={"ID":"38b428d7-ab0f-480d-bcfe-0779fb106abf","Type":"ContainerStarted","Data":"88065f9bfba95c7a22eb21eee7320a0466e8d7e4a40d621088e19659e5e7fef6"} Nov 02 20:24:24.750230 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.749970 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:24.751502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.751475 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" event={"ID":"944cd387-111f-4550-a4db-bae84d5fea95","Type":"ContainerStarted","Data":"f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9"} Nov 02 20:24:24.751646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.751528 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" event={"ID":"944cd387-111f-4550-a4db-bae84d5fea95","Type":"ContainerStarted","Data":"b3f4b78aab3dfa297afd7baa2ff6247fc43d2d375f835515a968b86a822c1ff7"} Nov 02 20:24:24.751646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.751564 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:24.856807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.856749 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" podStartSLOduration=1.8567345579999999 podStartE2EDuration="1.856734558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:24.855521999 +0000 UTC m=+2163.372294590" watchObservedRunningTime="2026-11-02 20:24:24.856734558 +0000 UTC m=+2163.373507144" Nov 02 20:24:24.964352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:24.964264 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" podStartSLOduration=1.9642467460000002 podStartE2EDuration="1.964246746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:24.962034697 +0000 UTC m=+2163.478807283" watchObservedRunningTime="2026-11-02 20:24:24.964246746 +0000 UTC m=+2163.481019332" Nov 02 20:24:35.761879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:35.761802 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:24:35.761879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:35.761873 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:24:45.800749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.800715 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z"] Nov 02 20:24:45.804639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.804618 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.867306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.862500 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6f1fd4-openshift-default-dockercfg-5t6rg\"" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.931622 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-token\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.931689 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-credential-socket\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.931732 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-data\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.931992 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfqq\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-kube-api-access-qxfqq\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.932132 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-certs\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.932211 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e54b-94b4-4453-b990-4a9a64c7d434-istiod-ca-cert\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.932311 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-socket\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.932485 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-podinfo\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.935311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.932528 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-envoy\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:45.963105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:45.943032 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z"] Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081058 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e54b-94b4-4453-b990-4a9a64c7d434-istiod-ca-cert\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081146 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-socket\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081212 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-podinfo\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081239 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-envoy\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081395 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-token\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081442 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-credential-socket\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081481 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-data\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081568 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfqq\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-kube-api-access-qxfqq\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.081608 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-certs\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.082158 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-certs\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.082866 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e54b-94b4-4453-b990-4a9a64c7d434-istiod-ca-cert\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.083116 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-socket\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.083753 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-credential-socket\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.084353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.084193 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-data\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.086957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.086932 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-envoy\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.088754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.088729 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-podinfo\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.155806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.155769 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfqq\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-kube-api-access-qxfqq\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.158905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.158854 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-token\") pod \"e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.264627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.264589 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5"] Nov 02 20:24:46.366601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.366502 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw"] Nov 02 20:24:46.420718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.420673 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:46.909055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.909026 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z"] Nov 02 20:24:46.909749 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:24:46.909715 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda882e54b_94b4_4453_b990_4a9a64c7d434.slice/crio-75e9e0523f5937bdd328fbf624ebc9953fe70e9c0cbc7e9a06de72062263f6aa WatchSource:0}: Error finding container 75e9e0523f5937bdd328fbf624ebc9953fe70e9c0cbc7e9a06de72062263f6aa: Status 404 returned error can't find the container with id 75e9e0523f5937bdd328fbf624ebc9953fe70e9c0cbc7e9a06de72062263f6aa Nov 02 20:24:46.912171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.912142 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:46.912258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.912207 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:46.912258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.912248 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:46.957465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:46.957435 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" event={"ID":"a882e54b-94b4-4453-b990-4a9a64c7d434","Type":"ContainerStarted","Data":"75e9e0523f5937bdd328fbf624ebc9953fe70e9c0cbc7e9a06de72062263f6aa"} Nov 02 20:24:47.966805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:47.966758 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" event={"ID":"a882e54b-94b4-4453-b990-4a9a64c7d434","Type":"ContainerStarted","Data":"353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c"} Nov 02 20:24:48.137412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:48.137351 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podStartSLOduration=3.13733327 podStartE2EDuration="3.13733327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:48.136410516 +0000 UTC m=+2186.653183102" watchObservedRunningTime="2026-11-02 20:24:48.13733327 +0000 UTC m=+2186.654105851" Nov 02 20:24:48.421382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:48.421340 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:48.438332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:48.438172 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:48.980847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:48.980813 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:48.982376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:48.982346 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:24:51.265535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.265485 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" podUID="ab607139-190f-4848-a5bb-bb62876c9bde" containerName="payload-processing" containerID="cri-o://5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8" gracePeriod=25 Nov 02 20:24:51.367158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.367090 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" podUID="dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" containerName="payload-pre-processing" containerID="cri-o://63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d" gracePeriod=25 Nov 02 20:24:51.544964 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:24:51.544905 5800 logging.go:55] [core] [Channel #4963 SubChannel #4964]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.175:50052", ServerName: "10.132.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.175:50052: connect: connection refused" Nov 02 20:24:51.588392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.588078 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:24:51.679360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.679328 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:24:51.691608 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.691580 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab607139-190f-4848-a5bb-bb62876c9bde-kube-api-access-452lz\" (UniqueName: \"kubernetes.io/projected/ab607139-190f-4848-a5bb-bb62876c9bde-kube-api-access-452lz\") pod \"ab607139-190f-4848-a5bb-bb62876c9bde\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " Nov 02 20:24:51.691829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.691809 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab607139-190f-4848-a5bb-bb62876c9bde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab607139-190f-4848-a5bb-bb62876c9bde-plugins-config-volume\") pod \"ab607139-190f-4848-a5bb-bb62876c9bde\" (UID: \"ab607139-190f-4848-a5bb-bb62876c9bde\") " Nov 02 20:24:51.692252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.692219 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab607139-190f-4848-a5bb-bb62876c9bde-plugins-config-volume" pod "ab607139-190f-4848-a5bb-bb62876c9bde" (UID: "ab607139-190f-4848-a5bb-bb62876c9bde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:51.693862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.693839 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab607139-190f-4848-a5bb-bb62876c9bde-kube-api-access-452lz" pod "ab607139-190f-4848-a5bb-bb62876c9bde" (UID: "ab607139-190f-4848-a5bb-bb62876c9bde"). InnerVolumeSpecName "kube-api-access-452lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:51.793114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.793028 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-kube-api-access-vzbf7\" (UniqueName: \"kubernetes.io/projected/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-kube-api-access-vzbf7\") pod \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " Nov 02 20:24:51.793319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.793275 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-plugins-config-volume\") pod \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\" (UID: \"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5\") " Nov 02 20:24:51.793717 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.793688 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-plugins-config-volume" pod "dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" (UID: "dc328d2b-6c7d-4ed2-b195-fb4659d7acc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:24:51.793824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.793723 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452lz\" (UniqueName: \"kubernetes.io/projected/ab607139-190f-4848-a5bb-bb62876c9bde-kube-api-access-452lz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:51.793824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.793739 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab607139-190f-4848-a5bb-bb62876c9bde-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:51.795133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.795106 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-kube-api-access-vzbf7" pod "dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" (UID: "dc328d2b-6c7d-4ed2-b195-fb4659d7acc5"). InnerVolumeSpecName "kube-api-access-vzbf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:24:51.895253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.895212 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:51.895253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:51.895253 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzbf7\" (UniqueName: \"kubernetes.io/projected/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5-kube-api-access-vzbf7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:24:52.003190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.003141 5800 generic.go:356] "Generic (PLEG): container finished" podID="ab607139-190f-4848-a5bb-bb62876c9bde" containerID="5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8" exitCode=0 Nov 02 20:24:52.004913 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.004889 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" Nov 02 20:24:52.007397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.007366 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" event={"ID":"ab607139-190f-4848-a5bb-bb62876c9bde","Type":"ContainerDied","Data":"5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8"} Nov 02 20:24:52.007584 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.007563 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5" event={"ID":"ab607139-190f-4848-a5bb-bb62876c9bde","Type":"ContainerDied","Data":"05000e530dfde80c7e1736b146de40a3c24a9202eca2ccb0833a75221fa03083"} Nov 02 20:24:52.007711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.007627 5800 scope.go:122] "RemoveContainer" containerID="5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8" Nov 02 20:24:52.009125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.009095 5800 generic.go:356] "Generic (PLEG): container finished" podID="dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" containerID="63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d" exitCode=0 Nov 02 20:24:52.009213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.009170 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" Nov 02 20:24:52.009271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.009208 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" event={"ID":"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5","Type":"ContainerDied","Data":"63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d"} Nov 02 20:24:52.009363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.009269 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" event={"ID":"dc328d2b-6c7d-4ed2-b195-fb4659d7acc5","Type":"ContainerDied","Data":"6c4a1c09fec60844614f99cb22a57dc0221e0b7c497dd24fd4198c45027d3337"} Nov 02 20:24:52.019493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.019467 5800 scope.go:122] "RemoveContainer" containerID="5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8" Nov 02 20:24:52.019811 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:24:52.019789 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8\": container with ID starting with 5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8 not found: ID does not exist" containerID="5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8" Nov 02 20:24:52.019893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.019817 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8"} err="failed to get container status \"5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8\": rpc error: code = NotFound desc = could not find container \"5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8\": container with ID starting with 5a9d9bfb81be2805e2541a28c5a85155b02e231b9fe74363840f86da8dcfdbe8 not found: ID does not exist" Nov 02 20:24:52.019893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.019835 5800 scope.go:122] "RemoveContainer" containerID="63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d" Nov 02 20:24:52.033015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.032990 5800 scope.go:122] "RemoveContainer" containerID="63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d" Nov 02 20:24:52.033380 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:24:52.033354 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d\": container with ID starting with 63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d not found: ID does not exist" containerID="63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d" Nov 02 20:24:52.033500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.033385 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d"} err="failed to get container status \"63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d\": rpc error: code = NotFound desc = could not find container \"63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d\": container with ID starting with 63a7e14aa4b398fa5feb6234ecd51144ed0c7bb28bed58d7564b0717623fb99d not found: ID does not exist" Nov 02 20:24:52.164653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.164560 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw"] Nov 02 20:24:52.169492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.169457 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw"] Nov 02 20:24:52.267069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.267028 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5"] Nov 02 20:24:52.275574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.275545 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-77fbb684f7-m99s5"] Nov 02 20:24:52.544520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:52.544470 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-6cf8fc8d67-6grfw" podUID="dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.175:50052\" within 1s: context deadline exceeded" Nov 02 20:24:52.698053 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:24:52.698002 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:24:53.623391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:53.623362 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab607139-190f-4848-a5bb-bb62876c9bde" path="/var/lib/kubelet/pods/ab607139-190f-4848-a5bb-bb62876c9bde/volumes" Nov 02 20:24:53.623815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:53.623783 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" path="/var/lib/kubelet/pods/dc328d2b-6c7d-4ed2-b195-fb4659d7acc5/volumes" Nov 02 20:24:54.067423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.067374 5800 generic.go:356] "Generic (PLEG): container finished" podID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerID="a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b" exitCode=0 Nov 02 20:24:54.067964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.067471 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" event={"ID":"7e78e4c5-6892-4e97-9f64-8e0c6da82a17","Type":"ContainerDied","Data":"a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b"} Nov 02 20:24:54.067964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.067931 5800 scope.go:122] "RemoveContainer" containerID="a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b" Nov 02 20:24:54.071343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.071308 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:54.071520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.071399 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:54.071520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.071445 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:24:54.602417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.602383 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr"] Nov 02 20:24:54.603066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.603037 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ab607139-190f-4848-a5bb-bb62876c9bde" containerName="payload-processing" Nov 02 20:24:54.603066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.603060 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab607139-190f-4848-a5bb-bb62876c9bde" containerName="payload-processing" Nov 02 20:24:54.603254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.603100 5800 cpu_manager.go:414] "containerMap: removing container" podUID="dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" containerName="payload-pre-processing" Nov 02 20:24:54.603254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.603106 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" containerName="payload-pre-processing" Nov 02 20:24:54.603254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.603208 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab607139-190f-4848-a5bb-bb62876c9bde" containerName="payload-processing" Nov 02 20:24:54.603254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.603225 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc328d2b-6c7d-4ed2-b195-fb4659d7acc5" containerName="payload-pre-processing" Nov 02 20:24:54.606764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.606743 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:54.656307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.655724 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6f1fd4\"" Nov 02 20:24:54.656307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.655872 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6f1fd4-dockercfg-rls87\"" Nov 02 20:24:54.662310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.662020 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr"] Nov 02 20:24:54.791577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.791535 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd836709-a27c-4734-93ba-6bd87fd7f9f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:54.791995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.791973 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bvrq\" (UniqueName: \"kubernetes.io/projected/cd836709-a27c-4734-93ba-6bd87fd7f9f8-kube-api-access-4bvrq\") pod \"payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:54.801476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.801443 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm"] Nov 02 20:24:54.814740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.814703 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:54.852858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.852768 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm"] Nov 02 20:24:54.893353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.893313 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140a2cc6-38b0-4eed-ae10-07be0897848e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:54.893546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.893370 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd836709-a27c-4734-93ba-6bd87fd7f9f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:54.893546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.893451 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h76hg\" (UniqueName: \"kubernetes.io/projected/140a2cc6-38b0-4eed-ae10-07be0897848e-kube-api-access-h76hg\") pod \"payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:54.893627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.893613 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bvrq\" (UniqueName: \"kubernetes.io/projected/cd836709-a27c-4734-93ba-6bd87fd7f9f8-kube-api-access-4bvrq\") pod \"payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:54.894820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.894796 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd836709-a27c-4734-93ba-6bd87fd7f9f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:54.961996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.961955 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bvrq\" (UniqueName: \"kubernetes.io/projected/cd836709-a27c-4734-93ba-6bd87fd7f9f8-kube-api-access-4bvrq\") pod \"payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:54.994891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.994852 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140a2cc6-38b0-4eed-ae10-07be0897848e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:54.995051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.994903 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76hg\" (UniqueName: \"kubernetes.io/projected/140a2cc6-38b0-4eed-ae10-07be0897848e-kube-api-access-h76hg\") pod \"payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:54.995695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:54.995672 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140a2cc6-38b0-4eed-ae10-07be0897848e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:55.063117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.063076 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h76hg\" (UniqueName: \"kubernetes.io/projected/140a2cc6-38b0-4eed-ae10-07be0897848e-kube-api-access-h76hg\") pod \"payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:55.079551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.079515 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" event={"ID":"7e78e4c5-6892-4e97-9f64-8e0c6da82a17","Type":"ContainerStarted","Data":"0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383"} Nov 02 20:24:55.162167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.162057 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:55.221091 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.221060 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:55.381169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.381135 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:24:55.383334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.382366 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 20:24:55.383334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.382467 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 20:24:55.588646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.588615 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm"] Nov 02 20:24:55.589789 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:24:55.589701 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140a2cc6_38b0_4eed_ae10_07be0897848e.slice/crio-e0640021149ceac28ab4e41667e2c7a24c7209248c495d7ef51f5eafc4ffc53a WatchSource:0}: Error finding container e0640021149ceac28ab4e41667e2c7a24c7209248c495d7ef51f5eafc4ffc53a: Status 404 returned error can't find the container with id e0640021149ceac28ab4e41667e2c7a24c7209248c495d7ef51f5eafc4ffc53a Nov 02 20:24:55.692272 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:24:55.691726 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd836709_a27c_4734_93ba_6bd87fd7f9f8.slice/crio-f19ec4a75c5193fc11fcd71910f1ab01add480c8265e96c9d5f13a90a569abf2 WatchSource:0}: Error finding container f19ec4a75c5193fc11fcd71910f1ab01add480c8265e96c9d5f13a90a569abf2: Status 404 returned error can't find the container with id f19ec4a75c5193fc11fcd71910f1ab01add480c8265e96c9d5f13a90a569abf2 Nov 02 20:24:55.692272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:55.692149 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr"] Nov 02 20:24:56.088115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.088076 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" event={"ID":"140a2cc6-38b0-4eed-ae10-07be0897848e","Type":"ContainerStarted","Data":"8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96"} Nov 02 20:24:56.088344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.088145 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:24:56.088344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.088168 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" event={"ID":"140a2cc6-38b0-4eed-ae10-07be0897848e","Type":"ContainerStarted","Data":"e0640021149ceac28ab4e41667e2c7a24c7209248c495d7ef51f5eafc4ffc53a"} Nov 02 20:24:56.089712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.089682 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" event={"ID":"cd836709-a27c-4734-93ba-6bd87fd7f9f8","Type":"ContainerStarted","Data":"7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8"} Nov 02 20:24:56.089831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.089735 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:24:56.089831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.089759 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" event={"ID":"cd836709-a27c-4734-93ba-6bd87fd7f9f8","Type":"ContainerStarted","Data":"f19ec4a75c5193fc11fcd71910f1ab01add480c8265e96c9d5f13a90a569abf2"} Nov 02 20:24:56.199034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.198984 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" podStartSLOduration=2.198967527 podStartE2EDuration="2.198967527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:56.195856142 +0000 UTC m=+2194.712628728" watchObservedRunningTime="2026-11-02 20:24:56.198967527 +0000 UTC m=+2194.715740134" Nov 02 20:24:56.304990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.303996 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" podStartSLOduration=2.303977002 podStartE2EDuration="2.303977002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:24:56.301624827 +0000 UTC m=+2194.818397425" watchObservedRunningTime="2026-11-02 20:24:56.303977002 +0000 UTC m=+2194.820749589" Nov 02 20:24:56.379601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.379500 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 20:24:56.379601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:56.379565 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 20:24:57.181262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:57.181227 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2"] Nov 02 20:24:57.181785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:57.181438 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" containerID="cri-o://c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4" gracePeriod=30 Nov 02 20:24:57.379484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:57.379449 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 20:24:57.379665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:57.379502 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 20:24:57.552307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:57.551466 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:24:58.379627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:58.379593 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 20:24:58.380020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:58.379646 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 20:24:59.379446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:59.379410 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Nov 02 20:24:59.379642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:24:59.379465 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Nov 02 20:25:00.446075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:00.445547 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:25:00.447972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:00.447938 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:25:02.453129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.453058 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:25:02.571595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.571558 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-envoy\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.571805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.571663 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f18a58-c218-4d0c-98a1-cc05dcc53725-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f18a58-c218-4d0c-98a1-cc05dcc53725-istiod-ca-cert\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.571805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.571696 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-socket\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.571805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.571748 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-data\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.571978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.571832 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-token\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-token\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.571978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.571904 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-credential-socket\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.571978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.571945 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-podinfo\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.572137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.572000 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-certs\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.572137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.572107 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-kube-api-access-p5rsm\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-kube-api-access-p5rsm\") pod \"97f18a58-c218-4d0c-98a1-cc05dcc53725\" (UID: \"97f18a58-c218-4d0c-98a1-cc05dcc53725\") " Nov 02 20:25:02.572249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.572181 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-data" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:02.572578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.572554 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f18a58-c218-4d0c-98a1-cc05dcc53725-istiod-ca-cert" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:02.572578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.572572 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.572817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.572797 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-socket" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:02.572990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.572973 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-credential-socket" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:02.573632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.573572 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-certs" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:02.575532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.575503 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-kube-api-access-p5rsm" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "kube-api-access-p5rsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:02.575942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.575880 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-envoy" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:25:02.576823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.576802 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-token" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:02.577042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.577017 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-podinfo" pod "97f18a58-c218-4d0c-98a1-cc05dcc53725" (UID: "97f18a58-c218-4d0c-98a1-cc05dcc53725"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:25:02.674041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674001 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rsm\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-kube-api-access-p5rsm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.674041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674046 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.674274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674061 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f18a58-c218-4d0c-98a1-cc05dcc53725-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.674274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674075 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.674274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674095 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.674274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674109 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.674274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674122 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f18a58-c218-4d0c-98a1-cc05dcc53725-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:02.674274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:02.674137 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f18a58-c218-4d0c-98a1-cc05dcc53725-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:03.143234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.143189 5800 generic.go:356] "Generic (PLEG): container finished" podID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerID="c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4" exitCode=0 Nov 02 20:25:03.143484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.143251 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" Nov 02 20:25:03.143484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.143316 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" event={"ID":"97f18a58-c218-4d0c-98a1-cc05dcc53725","Type":"ContainerDied","Data":"c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4"} Nov 02 20:25:03.143484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.143385 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2" event={"ID":"97f18a58-c218-4d0c-98a1-cc05dcc53725","Type":"ContainerDied","Data":"fdc93bb04fb0637c45bec8c2fe94fe6d86c6c1e2f18106b0176d845be1d02e69"} Nov 02 20:25:03.143484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.143416 5800 scope.go:122] "RemoveContainer" containerID="c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4" Nov 02 20:25:03.157498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.157480 5800 scope.go:122] "RemoveContainer" containerID="7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf" Nov 02 20:25:03.168439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.168415 5800 scope.go:122] "RemoveContainer" containerID="c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4" Nov 02 20:25:03.168748 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:25:03.168729 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4\": container with ID starting with c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4 not found: ID does not exist" containerID="c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4" Nov 02 20:25:03.168838 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.168759 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4"} err="failed to get container status \"c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4\": rpc error: code = NotFound desc = could not find container \"c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4\": container with ID starting with c74efb9a3a6c6d83228e86fe4c552dff15f7a3cf3441ef5c93e0eae03aec41d4 not found: ID does not exist" Nov 02 20:25:03.168838 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.168783 5800 scope.go:122] "RemoveContainer" containerID="7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf" Nov 02 20:25:03.169038 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:25:03.169021 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf\": container with ID starting with 7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf not found: ID does not exist" containerID="7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf" Nov 02 20:25:03.169094 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.169046 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf"} err="failed to get container status \"7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf\": rpc error: code = NotFound desc = could not find container \"7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf\": container with ID starting with 7dec0761a028db1ea139ec916235cd4798536de1ac390770ab5894d949c8cbbf not found: ID does not exist" Nov 02 20:25:03.198490 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:25:03.198462 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f18a58_c218_4d0c_98a1_cc05dcc53725.slice/crio-fdc93bb04fb0637c45bec8c2fe94fe6d86c6c1e2f18106b0176d845be1d02e69\": RecentStats: unable to find data in memory cache]" Nov 02 20:25:03.299268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.299231 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2"] Nov 02 20:25:03.345701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.345660 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c25349-openshift-default-695cf4f68f-8jhk2"] Nov 02 20:25:03.634652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:03.634609 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" path="/var/lib/kubelet/pods/97f18a58-c218-4d0c-98a1-cc05dcc53725/volumes" Nov 02 20:25:07.099795 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:07.099761 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:25:07.100236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:07.099828 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:25:22.048104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.048066 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn"] Nov 02 20:25:22.093047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.093006 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d405494c-f2kdn"] Nov 02 20:25:22.144664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.144631 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z"] Nov 02 20:25:22.144854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.144833 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" podUID="5b7169de-a109-4025-8ce8-af8bdcbaa1fa" containerName="payload-processing" containerID="cri-o://6332595fbb6fd1b800f8dc7842784f54874330d7d851d12c63bf56ac7f0ce263" gracePeriod=30 Nov 02 20:25:22.199897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.199863 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx"] Nov 02 20:25:22.200137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.200109 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" podUID="ca3e3d07-9a3f-4cd7-8363-9cbf30621371" containerName="payload-pre-processing" containerID="cri-o://e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00" gracePeriod=30 Nov 02 20:25:22.290434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.290392 5800 generic.go:356] "Generic (PLEG): container finished" podID="5b7169de-a109-4025-8ce8-af8bdcbaa1fa" containerID="6332595fbb6fd1b800f8dc7842784f54874330d7d851d12c63bf56ac7f0ce263" exitCode=0 Nov 02 20:25:22.290611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.290465 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" event={"ID":"5b7169de-a109-4025-8ce8-af8bdcbaa1fa","Type":"ContainerDied","Data":"6332595fbb6fd1b800f8dc7842784f54874330d7d851d12c63bf56ac7f0ce263"} Nov 02 20:25:22.494093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.494057 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:25:22.526058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.526032 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:25:22.619565 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.619479 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-kube-api-access-dh4dp\" (UniqueName: \"kubernetes.io/projected/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-kube-api-access-dh4dp\") pod \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " Nov 02 20:25:22.619752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.619607 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-plugins-config-volume\") pod \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " Nov 02 20:25:22.619752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.619717 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-plugins-config-volume\") pod \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\" (UID: \"5b7169de-a109-4025-8ce8-af8bdcbaa1fa\") " Nov 02 20:25:22.619877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.619802 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-kube-api-access-cf6r6\" (UniqueName: \"kubernetes.io/projected/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-kube-api-access-cf6r6\") pod \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\" (UID: \"ca3e3d07-9a3f-4cd7-8363-9cbf30621371\") " Nov 02 20:25:22.620071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.620039 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-plugins-config-volume" pod "ca3e3d07-9a3f-4cd7-8363-9cbf30621371" (UID: "ca3e3d07-9a3f-4cd7-8363-9cbf30621371"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:22.620071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.620061 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-plugins-config-volume" pod "5b7169de-a109-4025-8ce8-af8bdcbaa1fa" (UID: "5b7169de-a109-4025-8ce8-af8bdcbaa1fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:25:22.620914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.620378 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.620914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.620402 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.622205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.622183 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-kube-api-access-dh4dp" pod "5b7169de-a109-4025-8ce8-af8bdcbaa1fa" (UID: "5b7169de-a109-4025-8ce8-af8bdcbaa1fa"). InnerVolumeSpecName "kube-api-access-dh4dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:22.623227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.623208 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-kube-api-access-cf6r6" pod "ca3e3d07-9a3f-4cd7-8363-9cbf30621371" (UID: "ca3e3d07-9a3f-4cd7-8363-9cbf30621371"). InnerVolumeSpecName "kube-api-access-cf6r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:25:22.729089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.724183 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6r6\" (UniqueName: \"kubernetes.io/projected/ca3e3d07-9a3f-4cd7-8363-9cbf30621371-kube-api-access-cf6r6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.729089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:22.724217 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4dp\" (UniqueName: \"kubernetes.io/projected/5b7169de-a109-4025-8ce8-af8bdcbaa1fa-kube-api-access-dh4dp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:25:22.774068 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:25:22.774025 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:25:23.072459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.072426 5800 scope.go:122] "RemoveContainer" containerID="bc832d99da7b1a01ea4476292795053e47431339d37255c31baa775cc9884d51" Nov 02 20:25:23.082817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.082795 5800 scope.go:122] "RemoveContainer" containerID="1c9cc9192719c8f78a1cf4f118a5e3619b9a1068fd79bff3bb4df4699705336c" Nov 02 20:25:23.299807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.299759 5800 generic.go:356] "Generic (PLEG): container finished" podID="ca3e3d07-9a3f-4cd7-8363-9cbf30621371" containerID="e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00" exitCode=0 Nov 02 20:25:23.299979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.299898 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" event={"ID":"ca3e3d07-9a3f-4cd7-8363-9cbf30621371","Type":"ContainerDied","Data":"e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00"} Nov 02 20:25:23.299979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.299947 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" event={"ID":"ca3e3d07-9a3f-4cd7-8363-9cbf30621371","Type":"ContainerDied","Data":"fd80bfd54ee4c6fb3ca84166b0b3f607890af1789d48926422260ec6bb9d9818"} Nov 02 20:25:23.300076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.299985 5800 scope.go:122] "RemoveContainer" containerID="e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00" Nov 02 20:25:23.300136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.300123 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx" Nov 02 20:25:23.302059 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.302033 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" event={"ID":"5b7169de-a109-4025-8ce8-af8bdcbaa1fa","Type":"ContainerDied","Data":"6c2057a28aa15f6905149a35d519c83d0ec329608a17a0b60d02eab5476a7618"} Nov 02 20:25:23.302210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.302037 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z" Nov 02 20:25:23.311392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.311315 5800 scope.go:122] "RemoveContainer" containerID="e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00" Nov 02 20:25:23.312455 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:25:23.312432 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00\": container with ID starting with e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00 not found: ID does not exist" containerID="e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00" Nov 02 20:25:23.312553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.312460 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00"} err="failed to get container status \"e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00\": rpc error: code = NotFound desc = could not find container \"e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00\": container with ID starting with e01362bf0d5eaea6f979561c993621e426445a84625feb49928acba6dbe51a00 not found: ID does not exist" Nov 02 20:25:23.312553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.312478 5800 scope.go:122] "RemoveContainer" containerID="6332595fbb6fd1b800f8dc7842784f54874330d7d851d12c63bf56ac7f0ce263" Nov 02 20:25:23.453947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.453915 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx"] Nov 02 20:25:23.501977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.501940 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f83bc2-6f969fb6b7-wb5mx"] Nov 02 20:25:23.603983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.603884 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z"] Nov 02 20:25:23.609108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.609074 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f83bc2-74c68f44cd-kw27z"] Nov 02 20:25:23.625355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.625324 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9f1d7c-50c7-49c2-b909-0e76b04414a3" path="/var/lib/kubelet/pods/4c9f1d7c-50c7-49c2-b909-0e76b04414a3/volumes" Nov 02 20:25:23.625820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.625802 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7169de-a109-4025-8ce8-af8bdcbaa1fa" path="/var/lib/kubelet/pods/5b7169de-a109-4025-8ce8-af8bdcbaa1fa/volumes" Nov 02 20:25:23.626203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:23.626187 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3e3d07-9a3f-4cd7-8363-9cbf30621371" path="/var/lib/kubelet/pods/ca3e3d07-9a3f-4cd7-8363-9cbf30621371/volumes" Nov 02 20:25:46.973874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.973830 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh"] Nov 02 20:25:46.974639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974616 5800 cpu_manager.go:414] "containerMap: removing container" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" Nov 02 20:25:46.974743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974640 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" Nov 02 20:25:46.974743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974667 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ca3e3d07-9a3f-4cd7-8363-9cbf30621371" containerName="payload-pre-processing" Nov 02 20:25:46.974743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974675 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3e3d07-9a3f-4cd7-8363-9cbf30621371" containerName="payload-pre-processing" Nov 02 20:25:46.974743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974716 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5b7169de-a109-4025-8ce8-af8bdcbaa1fa" containerName="payload-processing" Nov 02 20:25:46.974743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974724 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7169de-a109-4025-8ce8-af8bdcbaa1fa" containerName="payload-processing" Nov 02 20:25:46.975006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974887 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" Nov 02 20:25:46.975006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974903 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7169de-a109-4025-8ce8-af8bdcbaa1fa" containerName="payload-processing" Nov 02 20:25:46.975006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974917 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" Nov 02 20:25:46.975006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.974930 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3e3d07-9a3f-4cd7-8363-9cbf30621371" containerName="payload-pre-processing" Nov 02 20:25:46.978334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:46.978313 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.040351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.040305 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh"] Nov 02 20:25:47.072560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.072458 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-57e2cf\"" Nov 02 20:25:47.072560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.072462 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-57e2cf-dockercfg-qzsr7\"" Nov 02 20:25:47.134623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.134589 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4qv\" (UniqueName: \"kubernetes.io/projected/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-kube-api-access-fx4qv\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.134804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.134636 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.236415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.236371 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4qv\" (UniqueName: \"kubernetes.io/projected/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-kube-api-access-fx4qv\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.236603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.236442 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.237409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.237383 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.289351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.289318 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4qv\" (UniqueName: \"kubernetes.io/projected/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-kube-api-access-fx4qv\") pod \"payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.291919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.291897 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:47.666542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.666512 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh"] Nov 02 20:25:47.745561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.745533 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf"] Nov 02 20:25:47.746466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.746445 5800 cpu_manager.go:414] "containerMap: removing container" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" Nov 02 20:25:47.746543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.746468 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f18a58-c218-4d0c-98a1-cc05dcc53725" containerName="istio-proxy" Nov 02 20:25:47.750491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.750470 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:47.799416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.799382 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf"] Nov 02 20:25:47.842328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.842264 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f04350c-cf11-4297-bcb0-9bc733c19a28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:47.842636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.842610 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcd4j\" (UniqueName: \"kubernetes.io/projected/4f04350c-cf11-4297-bcb0-9bc733c19a28-kube-api-access-jcd4j\") pod \"payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:47.943917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.943822 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcd4j\" (UniqueName: \"kubernetes.io/projected/4f04350c-cf11-4297-bcb0-9bc733c19a28-kube-api-access-jcd4j\") pod \"payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:47.943917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.943893 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f04350c-cf11-4297-bcb0-9bc733c19a28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:47.944935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.944912 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f04350c-cf11-4297-bcb0-9bc733c19a28-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:47.996622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:47.996586 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcd4j\" (UniqueName: \"kubernetes.io/projected/4f04350c-cf11-4297-bcb0-9bc733c19a28-kube-api-access-jcd4j\") pod \"payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:48.065939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:48.065900 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:48.434581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:48.434544 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf"] Nov 02 20:25:48.435777 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:25:48.435737 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f04350c_cf11_4297_bcb0_9bc733c19a28.slice/crio-5803196dd78f773946216ccc781606dfcd688e3754a85e6c96ea2b8a811a0818 WatchSource:0}: Error finding container 5803196dd78f773946216ccc781606dfcd688e3754a85e6c96ea2b8a811a0818: Status 404 returned error can't find the container with id 5803196dd78f773946216ccc781606dfcd688e3754a85e6c96ea2b8a811a0818 Nov 02 20:25:48.493430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:48.493395 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" event={"ID":"4f04350c-cf11-4297-bcb0-9bc733c19a28","Type":"ContainerStarted","Data":"5803196dd78f773946216ccc781606dfcd688e3754a85e6c96ea2b8a811a0818"} Nov 02 20:25:48.495064 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:48.495030 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" event={"ID":"173083ea-5a01-4fb6-94ff-7c63d8a6bd10","Type":"ContainerStarted","Data":"dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e"} Nov 02 20:25:48.495207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:48.495070 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" event={"ID":"173083ea-5a01-4fb6-94ff-7c63d8a6bd10","Type":"ContainerStarted","Data":"4d0483aecf40682a5314b24418a95ae614b97e984b97a62f08630aa674c2696e"} Nov 02 20:25:48.495207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:48.495162 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:25:48.602090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:48.602027 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" podStartSLOduration=2.602011693 podStartE2EDuration="2.602011693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:48.599912636 +0000 UTC m=+2247.116685222" watchObservedRunningTime="2026-11-02 20:25:48.602011693 +0000 UTC m=+2247.118784278" Nov 02 20:25:49.503743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:49.503706 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" event={"ID":"4f04350c-cf11-4297-bcb0-9bc733c19a28","Type":"ContainerStarted","Data":"10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567"} Nov 02 20:25:49.504225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:49.503838 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:25:49.611122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:49.611064 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" podStartSLOduration=2.611049833 podStartE2EDuration="2.611049833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:49.609101711 +0000 UTC m=+2248.125874298" watchObservedRunningTime="2026-11-02 20:25:49.611049833 +0000 UTC m=+2248.127822412" Nov 02 20:25:52.697330 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:25:52.697277 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:25:54.137883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.137849 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g"] Nov 02 20:25:54.142963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.142942 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.192519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.192483 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g"] Nov 02 20:25:54.234608 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.234577 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-408103-dockercfg-9drvd\"" Nov 02 20:25:54.234608 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.234582 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-408103\"" Nov 02 20:25:54.331516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.331485 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbppq\" (UniqueName: \"kubernetes.io/projected/db1276ca-35a0-4335-89ba-d44d3c06d9d5-kube-api-access-nbppq\") pod \"payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.331809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.331783 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1276ca-35a0-4335-89ba-d44d3c06d9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.432769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.432691 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1276ca-35a0-4335-89ba-d44d3c06d9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.432769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.432743 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbppq\" (UniqueName: \"kubernetes.io/projected/db1276ca-35a0-4335-89ba-d44d3c06d9d5-kube-api-access-nbppq\") pod \"payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.433448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.433426 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1276ca-35a0-4335-89ba-d44d3c06d9d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.486029 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.486000 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbppq\" (UniqueName: \"kubernetes.io/projected/db1276ca-35a0-4335-89ba-d44d3c06d9d5-kube-api-access-nbppq\") pod \"payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.759781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.759752 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:54.913430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.913390 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r"] Nov 02 20:25:54.917758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.917735 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:54.938178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.938153 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9d6f\" (UniqueName: \"kubernetes.io/projected/740e2f52-591a-4712-8898-a14aa59dd2a9-kube-api-access-t9d6f\") pod \"payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:54.938350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.938313 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740e2f52-591a-4712-8898-a14aa59dd2a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:54.969739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:54.969709 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r"] Nov 02 20:25:55.039085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.039005 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740e2f52-591a-4712-8898-a14aa59dd2a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:55.039236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.039137 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9d6f\" (UniqueName: \"kubernetes.io/projected/740e2f52-591a-4712-8898-a14aa59dd2a9-kube-api-access-t9d6f\") pod \"payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:55.039761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.039740 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740e2f52-591a-4712-8898-a14aa59dd2a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:55.099896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.099862 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9d6f\" (UniqueName: \"kubernetes.io/projected/740e2f52-591a-4712-8898-a14aa59dd2a9-kube-api-access-t9d6f\") pod \"payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:55.130331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.130293 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g"] Nov 02 20:25:55.135466 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:25:55.135421 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1276ca_35a0_4335_89ba_d44d3c06d9d5.slice/crio-c52d008427bb034ab6ab62a46e64f10e798e1697149634d7432fcce7492df537 WatchSource:0}: Error finding container c52d008427bb034ab6ab62a46e64f10e798e1697149634d7432fcce7492df537: Status 404 returned error can't find the container with id c52d008427bb034ab6ab62a46e64f10e798e1697149634d7432fcce7492df537 Nov 02 20:25:55.247691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.247648 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:55.554409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.554366 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" event={"ID":"db1276ca-35a0-4335-89ba-d44d3c06d9d5","Type":"ContainerStarted","Data":"bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee"} Nov 02 20:25:55.554616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.554486 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:25:55.554616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.554512 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" event={"ID":"db1276ca-35a0-4335-89ba-d44d3c06d9d5","Type":"ContainerStarted","Data":"c52d008427bb034ab6ab62a46e64f10e798e1697149634d7432fcce7492df537"} Nov 02 20:25:55.625621 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:25:55.625586 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740e2f52_591a_4712_8898_a14aa59dd2a9.slice/crio-8a4e51cc8777a9acabe2947e621183fa19c78446764ccae5781726fe4c30b06e WatchSource:0}: Error finding container 8a4e51cc8777a9acabe2947e621183fa19c78446764ccae5781726fe4c30b06e: Status 404 returned error can't find the container with id 8a4e51cc8777a9acabe2947e621183fa19c78446764ccae5781726fe4c30b06e Nov 02 20:25:55.627179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.627153 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r"] Nov 02 20:25:55.662362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:55.662273 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" podStartSLOduration=1.662254801 podStartE2EDuration="1.662254801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:55.662018959 +0000 UTC m=+2254.178791554" watchObservedRunningTime="2026-11-02 20:25:55.662254801 +0000 UTC m=+2254.179027387" Nov 02 20:25:56.565219 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:56.565171 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" event={"ID":"740e2f52-591a-4712-8898-a14aa59dd2a9","Type":"ContainerStarted","Data":"e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a"} Nov 02 20:25:56.565735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:56.565239 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" event={"ID":"740e2f52-591a-4712-8898-a14aa59dd2a9","Type":"ContainerStarted","Data":"8a4e51cc8777a9acabe2947e621183fa19c78446764ccae5781726fe4c30b06e"} Nov 02 20:25:56.565735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:56.565326 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:25:56.672264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:56.672088 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" podStartSLOduration=2.67206997 podStartE2EDuration="2.67206997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:56.671361025 +0000 UTC m=+2255.188133610" watchObservedRunningTime="2026-11-02 20:25:56.67206997 +0000 UTC m=+2255.188842555" Nov 02 20:25:59.506372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:25:59.506342 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:26:00.515045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:00.515014 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:26:06.568491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:06.568456 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:26:07.576728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:07.576698 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:26:11.016239 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.016204 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2"] Nov 02 20:26:11.016708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.016470 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" podUID="fad4472f-43aa-44b6-b2c0-8f57814488e9" containerName="payload-processing" containerID="cri-o://736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e" gracePeriod=30 Nov 02 20:26:11.114882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.114841 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8"] Nov 02 20:26:11.115099 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.115060 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" podUID="86d7073d-22ad-44d5-b02d-4c5e27728c40" containerName="payload-pre-processing" containerID="cri-o://ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba" gracePeriod=30 Nov 02 20:26:11.345013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.344985 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:26:11.400580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.400554 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:26:11.461762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.461732 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad4472f-43aa-44b6-b2c0-8f57814488e9-kube-api-access-nprpj\" (UniqueName: \"kubernetes.io/projected/fad4472f-43aa-44b6-b2c0-8f57814488e9-kube-api-access-nprpj\") pod \"fad4472f-43aa-44b6-b2c0-8f57814488e9\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " Nov 02 20:26:11.461976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.461873 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad4472f-43aa-44b6-b2c0-8f57814488e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4472f-43aa-44b6-b2c0-8f57814488e9-plugins-config-volume\") pod \"fad4472f-43aa-44b6-b2c0-8f57814488e9\" (UID: \"fad4472f-43aa-44b6-b2c0-8f57814488e9\") " Nov 02 20:26:11.462346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.462309 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad4472f-43aa-44b6-b2c0-8f57814488e9-plugins-config-volume" pod "fad4472f-43aa-44b6-b2c0-8f57814488e9" (UID: "fad4472f-43aa-44b6-b2c0-8f57814488e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:11.462523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.462498 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad4472f-43aa-44b6-b2c0-8f57814488e9-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:11.464533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.464505 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad4472f-43aa-44b6-b2c0-8f57814488e9-kube-api-access-nprpj" pod "fad4472f-43aa-44b6-b2c0-8f57814488e9" (UID: "fad4472f-43aa-44b6-b2c0-8f57814488e9"). InnerVolumeSpecName "kube-api-access-nprpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:11.580354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.573199 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d7073d-22ad-44d5-b02d-4c5e27728c40-kube-api-access-67z6d\" (UniqueName: \"kubernetes.io/projected/86d7073d-22ad-44d5-b02d-4c5e27728c40-kube-api-access-67z6d\") pod \"86d7073d-22ad-44d5-b02d-4c5e27728c40\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " Nov 02 20:26:11.580354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.573484 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d7073d-22ad-44d5-b02d-4c5e27728c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7073d-22ad-44d5-b02d-4c5e27728c40-plugins-config-volume\") pod \"86d7073d-22ad-44d5-b02d-4c5e27728c40\" (UID: \"86d7073d-22ad-44d5-b02d-4c5e27728c40\") " Nov 02 20:26:11.580354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.573941 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprpj\" (UniqueName: \"kubernetes.io/projected/fad4472f-43aa-44b6-b2c0-8f57814488e9-kube-api-access-nprpj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:11.580632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.580409 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d7073d-22ad-44d5-b02d-4c5e27728c40-plugins-config-volume" pod "86d7073d-22ad-44d5-b02d-4c5e27728c40" (UID: "86d7073d-22ad-44d5-b02d-4c5e27728c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:11.607945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.607906 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d7073d-22ad-44d5-b02d-4c5e27728c40-kube-api-access-67z6d" pod "86d7073d-22ad-44d5-b02d-4c5e27728c40" (UID: "86d7073d-22ad-44d5-b02d-4c5e27728c40"). InnerVolumeSpecName "kube-api-access-67z6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:11.675723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.675688 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67z6d\" (UniqueName: \"kubernetes.io/projected/86d7073d-22ad-44d5-b02d-4c5e27728c40-kube-api-access-67z6d\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:11.675723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.675718 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7073d-22ad-44d5-b02d-4c5e27728c40-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:11.681685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.681657 5800 generic.go:356] "Generic (PLEG): container finished" podID="fad4472f-43aa-44b6-b2c0-8f57814488e9" containerID="736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e" exitCode=0 Nov 02 20:26:11.681827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.681700 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" Nov 02 20:26:11.681827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.681742 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" event={"ID":"fad4472f-43aa-44b6-b2c0-8f57814488e9","Type":"ContainerDied","Data":"736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e"} Nov 02 20:26:11.681827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.681794 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2" event={"ID":"fad4472f-43aa-44b6-b2c0-8f57814488e9","Type":"ContainerDied","Data":"25c6e1e5589922c235fe8d97de30d58da3c46e5d958ea0ab569ad0e24875c313"} Nov 02 20:26:11.681827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.681822 5800 scope.go:122] "RemoveContainer" containerID="736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e" Nov 02 20:26:11.683255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.683231 5800 generic.go:356] "Generic (PLEG): container finished" podID="86d7073d-22ad-44d5-b02d-4c5e27728c40" containerID="ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba" exitCode=0 Nov 02 20:26:11.683384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.683267 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" event={"ID":"86d7073d-22ad-44d5-b02d-4c5e27728c40","Type":"ContainerDied","Data":"ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba"} Nov 02 20:26:11.683384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.683321 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" Nov 02 20:26:11.683384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.683327 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8" event={"ID":"86d7073d-22ad-44d5-b02d-4c5e27728c40","Type":"ContainerDied","Data":"bceda30a278c2aa65b851d163a337564e286bdf0d76119e36c685b73049dccf2"} Nov 02 20:26:11.692273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.692250 5800 scope.go:122] "RemoveContainer" containerID="736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e" Nov 02 20:26:11.692643 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:11.692617 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e\": container with ID starting with 736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e not found: ID does not exist" containerID="736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e" Nov 02 20:26:11.692748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.692649 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e"} err="failed to get container status \"736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e\": rpc error: code = NotFound desc = could not find container \"736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e\": container with ID starting with 736d997e8c20dca8064111c1e6637ace23a3606322e604677016f604653cbe7e not found: ID does not exist" Nov 02 20:26:11.692748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.692668 5800 scope.go:122] "RemoveContainer" containerID="ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba" Nov 02 20:26:11.702845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.702821 5800 scope.go:122] "RemoveContainer" containerID="ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba" Nov 02 20:26:11.703197 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:11.703171 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba\": container with ID starting with ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba not found: ID does not exist" containerID="ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba" Nov 02 20:26:11.703316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.703203 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba"} err="failed to get container status \"ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba\": rpc error: code = NotFound desc = could not find container \"ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba\": container with ID starting with ec3be74ebd4705effef5abd8c273d922062da35e212dabd58cdc6cfe650229ba not found: ID does not exist" Nov 02 20:26:11.836599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.836499 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2"] Nov 02 20:26:11.839803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.839777 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3b8db2-f558599d-zg7g2"] Nov 02 20:26:11.939882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.939840 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8"] Nov 02 20:26:11.944310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:11.944266 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3b8db2-55cf66cd8c-ts8c8"] Nov 02 20:26:13.624409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:13.624380 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d7073d-22ad-44d5-b02d-4c5e27728c40" path="/var/lib/kubelet/pods/86d7073d-22ad-44d5-b02d-4c5e27728c40/volumes" Nov 02 20:26:13.624823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:13.624801 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad4472f-43aa-44b6-b2c0-8f57814488e9" path="/var/lib/kubelet/pods/fad4472f-43aa-44b6-b2c0-8f57814488e9/volumes" Nov 02 20:26:16.721159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.721126 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb"] Nov 02 20:26:16.721871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.721851 5800 cpu_manager.go:414] "containerMap: removing container" podUID="86d7073d-22ad-44d5-b02d-4c5e27728c40" containerName="payload-pre-processing" Nov 02 20:26:16.721962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.721872 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d7073d-22ad-44d5-b02d-4c5e27728c40" containerName="payload-pre-processing" Nov 02 20:26:16.721962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.721918 5800 cpu_manager.go:414] "containerMap: removing container" podUID="fad4472f-43aa-44b6-b2c0-8f57814488e9" containerName="payload-processing" Nov 02 20:26:16.721962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.721924 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad4472f-43aa-44b6-b2c0-8f57814488e9" containerName="payload-processing" Nov 02 20:26:16.722133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.722023 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d7073d-22ad-44d5-b02d-4c5e27728c40" containerName="payload-pre-processing" Nov 02 20:26:16.722133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.722038 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad4472f-43aa-44b6-b2c0-8f57814488e9" containerName="payload-processing" Nov 02 20:26:16.725509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.725490 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.828733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.828700 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb"] Nov 02 20:26:16.844646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844607 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-token\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844658 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-podinfo\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844702 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istiod-ca-cert\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844757 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-socket\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844784 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-credential-socket\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844832 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-envoy\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844864 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-data\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844889 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnvw\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-kube-api-access-bsnvw\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.844996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.844947 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-certs\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.872549 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.863344 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7cad76-openshift-default-dockercfg-nrb75\"" Nov 02 20:26:16.946037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.945996 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-token\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946043 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-podinfo\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946088 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istiod-ca-cert\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946143 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-socket\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946174 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-credential-socket\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946220 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-envoy\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946361 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-data\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946389 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnvw\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-kube-api-access-bsnvw\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946429 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-certs\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946650 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-socket\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946693 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-credential-socket\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.946898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.946874 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istiod-ca-cert\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.947135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.947113 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-data\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.947177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.947156 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-certs\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.948667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.948633 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-envoy\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.949014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.948991 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-podinfo\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:16.999771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:16.999740 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnvw\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-kube-api-access-bsnvw\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:17.000090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.000070 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-token\") pod \"e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:17.039204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.039156 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:17.509149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.509120 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb"] Nov 02 20:26:17.509882 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:17.509855 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b0d508_997e_4a26_9fa6_f02b398ac7ce.slice/crio-e647db3078fb657bd73f3381f8a6d2393308a2c680c1f0ede89cda18f3c55fc9 WatchSource:0}: Error finding container e647db3078fb657bd73f3381f8a6d2393308a2c680c1f0ede89cda18f3c55fc9: Status 404 returned error can't find the container with id e647db3078fb657bd73f3381f8a6d2393308a2c680c1f0ede89cda18f3c55fc9 Nov 02 20:26:17.512397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.512364 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:17.512498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.512429 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:17.512498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.512457 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:17.729474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.729435 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" event={"ID":"51b0d508-997e-4a26-9fa6-f02b398ac7ce","Type":"ContainerStarted","Data":"dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9"} Nov 02 20:26:17.729964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.729503 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" event={"ID":"51b0d508-997e-4a26-9fa6-f02b398ac7ce","Type":"ContainerStarted","Data":"e647db3078fb657bd73f3381f8a6d2393308a2c680c1f0ede89cda18f3c55fc9"} Nov 02 20:26:17.844692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:17.844572 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" podStartSLOduration=1.84455307 podStartE2EDuration="1.84455307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:17.838688543 +0000 UTC m=+2276.355461125" watchObservedRunningTime="2026-11-02 20:26:17.84455307 +0000 UTC m=+2276.361325656" Nov 02 20:26:18.040036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:18.040000 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:18.046135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:18.046063 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:18.738041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:18.738009 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:18.739702 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:18.739683 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:26:19.215139 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.215081 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd"] Nov 02 20:26:19.215476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.215416 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" containerID="cri-o://9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c" gracePeriod=30 Nov 02 20:26:19.218223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.218201 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw"] Nov 02 20:26:19.218442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.218412 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" containerID="cri-o://b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c" gracePeriod=30 Nov 02 20:26:19.261962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.261919 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q"] Nov 02 20:26:19.262175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.262153 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" containerID="cri-o://1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08" gracePeriod=30 Nov 02 20:26:19.264812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.264781 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg"] Nov 02 20:26:19.265004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.264971 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" containerID="cri-o://13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e" gracePeriod=30 Nov 02 20:26:19.485540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.485463 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:26:19.485540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.485535 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:26:19.613378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.613332 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:26:19.613560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:19.613403 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:26:22.696133 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:22.696080 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:26:23.130309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:23.130262 5800 scope.go:122] "RemoveContainer" containerID="304070b0a7d532e738e6834e72ff8104226d92f769e75bc2d163ff9a70d208de" Nov 02 20:26:24.590035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.590005 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:26:24.624991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.624957 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:26:24.639939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.639908 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee123445-e0bb-4395-a5ce-0829b4288d3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee123445-e0bb-4395-a5ce-0829b4288d3d-istiod-ca-cert\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.639962 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-envoy\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640005 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-data\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640123 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-token\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-token\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640177 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-kube-api-access-pztm9\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-kube-api-access-pztm9\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640386 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640227 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-certs\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640386 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640329 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-socket\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640383 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-credential-socket\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.640497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640271 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee123445-e0bb-4395-a5ce-0829b4288d3d-istiod-ca-cert" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:24.640497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640422 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-podinfo\") pod \"ee123445-e0bb-4395-a5ce-0829b4288d3d\" (UID: \"ee123445-e0bb-4395-a5ce-0829b4288d3d\") " Nov 02 20:26:24.641559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.640926 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-socket" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.641559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.641145 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.641559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.641165 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee123445-e0bb-4395-a5ce-0829b4288d3d-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.641559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.641273 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-certs" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.641559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.641454 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-credential-socket" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.641951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.641789 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-data" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.642996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.642961 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:26:24.644054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.644024 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-podinfo" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:26:24.647357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.647330 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 20:26:24.648563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.648541 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-token" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.648667 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.648650 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-envoy" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.650140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.650120 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-kube-api-access-pztm9" pod "ee123445-e0bb-4395-a5ce-0829b4288d3d" (UID: "ee123445-e0bb-4395-a5ce-0829b4288d3d"). InnerVolumeSpecName "kube-api-access-pztm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.741974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.741940 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-kube-api-access-kq8nt\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-kube-api-access-kq8nt\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.741984 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-podinfo\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.742221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742075 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284b06b7-be99-435c-a148-0bbe1620cb50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b06b7-be99-435c-a148-0bbe1620cb50-istiod-ca-cert\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.742221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742127 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-envoy\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.742221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742170 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-podinfo\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742198 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-socket\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.742572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742319 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istiod-ca-cert\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742371 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-data\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.742572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742415 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-istio-token\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-istio-token\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.742572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742460 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-credential-socket\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.742572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742501 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-certs\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742539 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/284b06b7-be99-435c-a148-0bbe1620cb50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b06b7-be99-435c-a148-0bbe1620cb50-istio-podinfo\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742593 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-credential-socket\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742626 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-certs\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742662 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-envoy\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742697 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-envoy\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742728 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-data\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742764 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-credential-socket\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742794 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-token\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-token\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.742904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742897 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-certs\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.743346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742935 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-data\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.743346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.742997 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-socket\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.743346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743036 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-socket\") pod \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\" (UID: \"6b65472e-f834-406f-ae97-d4aecaa5e1e4\") " Nov 02 20:26:24.743346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743093 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63815b6-e9af-4d31-8b8d-c309561f8433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63815b6-e9af-4d31-8b8d-c309561f8433-istiod-ca-cert\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.743346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743122 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-token\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-token\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.743346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743158 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-kube-api-access-l5p2k\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-kube-api-access-l5p2k\") pod \"284b06b7-be99-435c-a148-0bbe1620cb50\" (UID: \"284b06b7-be99-435c-a148-0bbe1620cb50\") " Nov 02 20:26:24.743346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743194 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-kube-api-access-6k7wj\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-kube-api-access-6k7wj\") pod \"e63815b6-e9af-4d31-8b8d-c309561f8433\" (UID: \"e63815b6-e9af-4d31-8b8d-c309561f8433\") " Nov 02 20:26:24.743737 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743511 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284b06b7-be99-435c-a148-0bbe1620cb50-istiod-ca-cert" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743956 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743983 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b06b7-be99-435c-a148-0bbe1620cb50-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.743999 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.744014 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.744029 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.744043 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pztm9\" (UniqueName: \"kubernetes.io/projected/ee123445-e0bb-4395-a5ce-0829b4288d3d-kube-api-access-pztm9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.744059 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.744074 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee123445-e0bb-4395-a5ce-0829b4288d3d-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.744683 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-data" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.745009 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-socket" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.745346 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istiod-ca-cert" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:24.745941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.745604 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-credential-socket" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.748976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.746257 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-kube-api-access-6k7wj" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "kube-api-access-6k7wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.748976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.746589 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-certs" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.748976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.746851 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-data" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.748976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.747015 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-socket" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.748976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.747173 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-socket" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.748976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.747260 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-certs" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.748976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.748041 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63815b6-e9af-4d31-8b8d-c309561f8433-istiod-ca-cert" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:26:24.754486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.754456 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-token" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.754666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.754644 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-token" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.754735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.754702 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-istio-token" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.754802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.754767 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-envoy" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.754859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.754813 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/284b06b7-be99-435c-a148-0bbe1620cb50-istio-podinfo" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:26:24.755120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755089 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-podinfo" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:26:24.755353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755328 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-credential-socket" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.755353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755269 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-data" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.755525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755504 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-credential-socket" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.755525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755512 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-certs" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.755674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755570 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-kube-api-access-kq8nt" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "kube-api-access-kq8nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.755903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755863 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-envoy" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.755993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.755944 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-podinfo" pod "e63815b6-e9af-4d31-8b8d-c309561f8433" (UID: "e63815b6-e9af-4d31-8b8d-c309561f8433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:26:24.756572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.756547 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-envoy" pod "6b65472e-f834-406f-ae97-d4aecaa5e1e4" (UID: "6b65472e-f834-406f-ae97-d4aecaa5e1e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:26:24.757647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.757625 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-kube-api-access-l5p2k" pod "284b06b7-be99-435c-a148-0bbe1620cb50" (UID: "284b06b7-be99-435c-a148-0bbe1620cb50"). InnerVolumeSpecName "kube-api-access-l5p2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:24.787518 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.787486 5800 generic.go:356] "Generic (PLEG): container finished" podID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerID="1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08" exitCode=0 Nov 02 20:26:24.787696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.787536 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" Nov 02 20:26:24.787696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.787568 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" event={"ID":"6b65472e-f834-406f-ae97-d4aecaa5e1e4","Type":"ContainerDied","Data":"1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08"} Nov 02 20:26:24.787696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.787624 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q" event={"ID":"6b65472e-f834-406f-ae97-d4aecaa5e1e4","Type":"ContainerDied","Data":"e3d8fa656b9c3d9188812879e3867bd3f5dc8f2ef3dcb2f500dbad687c4ab4c0"} Nov 02 20:26:24.787696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.787658 5800 scope.go:122] "RemoveContainer" containerID="1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08" Nov 02 20:26:24.789149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.789123 5800 generic.go:356] "Generic (PLEG): container finished" podID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerID="9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c" exitCode=0 Nov 02 20:26:24.789271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.789181 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" Nov 02 20:26:24.789271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.789213 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" event={"ID":"e63815b6-e9af-4d31-8b8d-c309561f8433","Type":"ContainerDied","Data":"9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c"} Nov 02 20:26:24.789271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.789262 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd" event={"ID":"e63815b6-e9af-4d31-8b8d-c309561f8433","Type":"ContainerDied","Data":"318fd2da68602dd8c07165427cde4702ca9f5d1b2971b4fcf7a12b791d4880cb"} Nov 02 20:26:24.790907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.790884 5800 generic.go:356] "Generic (PLEG): container finished" podID="284b06b7-be99-435c-a148-0bbe1620cb50" containerID="b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c" exitCode=0 Nov 02 20:26:24.791018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.790910 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" event={"ID":"284b06b7-be99-435c-a148-0bbe1620cb50","Type":"ContainerDied","Data":"b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c"} Nov 02 20:26:24.791018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.790927 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" Nov 02 20:26:24.791018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.790960 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw" event={"ID":"284b06b7-be99-435c-a148-0bbe1620cb50","Type":"ContainerDied","Data":"633d323529c16fac0835730d2ba2a1078944966567f0261cae6d3f228bccdc72"} Nov 02 20:26:24.792628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.792605 5800 generic.go:356] "Generic (PLEG): container finished" podID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerID="13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e" exitCode=0 Nov 02 20:26:24.792730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.792632 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" event={"ID":"ee123445-e0bb-4395-a5ce-0829b4288d3d","Type":"ContainerDied","Data":"13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e"} Nov 02 20:26:24.792730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.792647 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" Nov 02 20:26:24.792730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.792694 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg" event={"ID":"ee123445-e0bb-4395-a5ce-0829b4288d3d","Type":"ContainerDied","Data":"fd8bd451df43458d8e1d475e3dcfcb8b2efb7c01797b9d66aaa02ef9f5d7f701"} Nov 02 20:26:24.799830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.799806 5800 scope.go:122] "RemoveContainer" containerID="1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08" Nov 02 20:26:24.800153 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:24.800125 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08\": container with ID starting with 1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08 not found: ID does not exist" containerID="1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08" Nov 02 20:26:24.800235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.800164 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08"} err="failed to get container status \"1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08\": rpc error: code = NotFound desc = could not find container \"1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08\": container with ID starting with 1703b411d4accdf2d05712b0a04955a548159df88b19a4a3d44112c88f3a0b08 not found: ID does not exist" Nov 02 20:26:24.800235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.800191 5800 scope.go:122] "RemoveContainer" containerID="9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c" Nov 02 20:26:24.810834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.810817 5800 scope.go:122] "RemoveContainer" containerID="9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c" Nov 02 20:26:24.811082 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:24.811064 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c\": container with ID starting with 9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c not found: ID does not exist" containerID="9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c" Nov 02 20:26:24.811143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.811089 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c"} err="failed to get container status \"9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c\": rpc error: code = NotFound desc = could not find container \"9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c\": container with ID starting with 9e6cdf63623b2451c69d9d623a69db79b7487d26970ae36c7f7b8bb4fdc1aa8c not found: ID does not exist" Nov 02 20:26:24.811143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.811103 5800 scope.go:122] "RemoveContainer" containerID="b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c" Nov 02 20:26:24.821992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.821970 5800 scope.go:122] "RemoveContainer" containerID="b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c" Nov 02 20:26:24.822829 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:24.822786 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c\": container with ID starting with b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c not found: ID does not exist" containerID="b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c" Nov 02 20:26:24.822932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.822836 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c"} err="failed to get container status \"b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c\": rpc error: code = NotFound desc = could not find container \"b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c\": container with ID starting with b0f2ae794c060471235fd3ca4901ef458318469ab27b38f94f9a48a163d3207c not found: ID does not exist" Nov 02 20:26:24.822932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.822866 5800 scope.go:122] "RemoveContainer" containerID="13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e" Nov 02 20:26:24.835243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.835161 5800 scope.go:122] "RemoveContainer" containerID="13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e" Nov 02 20:26:24.835737 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:24.835716 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e\": container with ID starting with 13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e not found: ID does not exist" containerID="13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e" Nov 02 20:26:24.835828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.835743 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e"} err="failed to get container status \"13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e\": rpc error: code = NotFound desc = could not find container \"13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e\": container with ID starting with 13070a6890ee71b4bb1ee28dfe4838c6bdf06539ef9bd2c193f901f1455d706e not found: ID does not exist" Nov 02 20:26:24.845814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845786 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845818 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845836 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845850 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845863 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845879 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b06b7-be99-435c-a148-0bbe1620cb50-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845892 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845906 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.845929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845920 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845933 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845948 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845962 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845975 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.845991 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846005 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846020 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846034 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65472e-f834-406f-ae97-d4aecaa5e1e4-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846048 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e63815b6-e9af-4d31-8b8d-c309561f8433-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846060 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846076 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5p2k\" (UniqueName: \"kubernetes.io/projected/284b06b7-be99-435c-a148-0bbe1620cb50-kube-api-access-l5p2k\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846099 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k7wj\" (UniqueName: \"kubernetes.io/projected/e63815b6-e9af-4d31-8b8d-c309561f8433-kube-api-access-6k7wj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846113 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq8nt\" (UniqueName: \"kubernetes.io/projected/6b65472e-f834-406f-ae97-d4aecaa5e1e4-kube-api-access-kq8nt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846128 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846142 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e63815b6-e9af-4d31-8b8d-c309561f8433-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846155 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65472e-f834-406f-ae97-d4aecaa5e1e4-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:24.846352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:24.846169 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b06b7-be99-435c-a148-0bbe1620cb50-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:25.143468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.143377 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd"] Nov 02 20:26:25.148185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.148151 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-vxdqd"] Nov 02 20:26:25.247464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.247432 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg"] Nov 02 20:26:25.250427 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.250401 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-dk9wg"] Nov 02 20:26:25.351916 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.351879 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw"] Nov 02 20:26:25.356423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.356393 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-jb7jw"] Nov 02 20:26:25.458694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.458612 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q"] Nov 02 20:26:25.461827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.461803 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9vr7q"] Nov 02 20:26:25.645306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.644690 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" path="/var/lib/kubelet/pods/284b06b7-be99-435c-a148-0bbe1620cb50/volumes" Nov 02 20:26:25.677390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.672077 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" path="/var/lib/kubelet/pods/6b65472e-f834-406f-ae97-d4aecaa5e1e4/volumes" Nov 02 20:26:25.677390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.672833 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" path="/var/lib/kubelet/pods/e63815b6-e9af-4d31-8b8d-c309561f8433/volumes" Nov 02 20:26:25.677390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:25.673477 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" path="/var/lib/kubelet/pods/ee123445-e0bb-4395-a5ce-0829b4288d3d/volumes" Nov 02 20:26:26.494419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.494383 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9"] Nov 02 20:26:26.494976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.494963 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" Nov 02 20:26:26.495026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.494977 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" Nov 02 20:26:26.495026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.494999 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" Nov 02 20:26:26.495026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495004 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" Nov 02 20:26:26.495026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495020 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" Nov 02 20:26:26.495026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495027 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" Nov 02 20:26:26.495183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495045 5800 cpu_manager.go:414] "containerMap: removing container" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" Nov 02 20:26:26.495183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495050 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" Nov 02 20:26:26.495183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495133 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63815b6-e9af-4d31-8b8d-c309561f8433" containerName="istio-proxy" Nov 02 20:26:26.495183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495144 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee123445-e0bb-4395-a5ce-0829b4288d3d" containerName="istio-proxy" Nov 02 20:26:26.495183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495153 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="284b06b7-be99-435c-a148-0bbe1620cb50" containerName="istio-proxy" Nov 02 20:26:26.495183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.495162 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b65472e-f834-406f-ae97-d4aecaa5e1e4" containerName="istio-proxy" Nov 02 20:26:26.500492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.500470 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.552004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.551599 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7cad76\"" Nov 02 20:26:26.558687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.558298 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9"] Nov 02 20:26:26.570608 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.570574 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7cad76-dockercfg-c99b9\"" Nov 02 20:26:26.571149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.571013 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.571149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.571095 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw728\" (UniqueName: \"kubernetes.io/projected/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-kube-api-access-sw728\") pod \"payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.663831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.663794 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27"] Nov 02 20:26:26.668219 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.668194 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:26.672534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.672502 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.672957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.672925 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw728\" (UniqueName: \"kubernetes.io/projected/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-kube-api-access-sw728\") pod \"payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.673408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.673389 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.721345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.721308 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27"] Nov 02 20:26:26.725808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.725780 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw728\" (UniqueName: \"kubernetes.io/projected/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-kube-api-access-sw728\") pod \"payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.774139 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.774100 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:26.774374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.774269 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm97n\" (UniqueName: \"kubernetes.io/projected/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-kube-api-access-pm97n\") pod \"payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:26.814936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.814899 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:26.878037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.877961 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:26.878237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.878079 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm97n\" (UniqueName: \"kubernetes.io/projected/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-kube-api-access-pm97n\") pod \"payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:26.881594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.879690 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:26.932004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.931966 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm97n\" (UniqueName: \"kubernetes.io/projected/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-kube-api-access-pm97n\") pod \"payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:26.998241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:26.997754 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:27.201572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.201526 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9"] Nov 02 20:26:27.208013 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:27.207975 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5220a8_dfa9_4597_9d6c_4b307bf78a64.slice/crio-8ab34e310f2f3ce8e5904c7aee7dbecb8a289756ba9b98a8fc6a5c353b83db28 WatchSource:0}: Error finding container 8ab34e310f2f3ce8e5904c7aee7dbecb8a289756ba9b98a8fc6a5c353b83db28: Status 404 returned error can't find the container with id 8ab34e310f2f3ce8e5904c7aee7dbecb8a289756ba9b98a8fc6a5c353b83db28 Nov 02 20:26:27.383004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.382970 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27"] Nov 02 20:26:27.383700 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:27.383668 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3c76e3_a6d1_40d8_80a3_b4dda1ea0eff.slice/crio-a8d0a43a8e959b4bc4f027372757a863bdc590dce449b21270e8868d887c9d87 WatchSource:0}: Error finding container a8d0a43a8e959b4bc4f027372757a863bdc590dce449b21270e8868d887c9d87: Status 404 returned error can't find the container with id a8d0a43a8e959b4bc4f027372757a863bdc590dce449b21270e8868d887c9d87 Nov 02 20:26:27.819295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.819231 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" event={"ID":"1a5220a8-dfa9-4597-9d6c-4b307bf78a64","Type":"ContainerStarted","Data":"1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504"} Nov 02 20:26:27.819743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.819328 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" event={"ID":"1a5220a8-dfa9-4597-9d6c-4b307bf78a64","Type":"ContainerStarted","Data":"8ab34e310f2f3ce8e5904c7aee7dbecb8a289756ba9b98a8fc6a5c353b83db28"} Nov 02 20:26:27.819743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.819608 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:27.821000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.820971 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" event={"ID":"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff","Type":"ContainerStarted","Data":"1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2"} Nov 02 20:26:27.821133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.821016 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" event={"ID":"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff","Type":"ContainerStarted","Data":"a8d0a43a8e959b4bc4f027372757a863bdc590dce449b21270e8868d887c9d87"} Nov 02 20:26:27.821133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.821046 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:27.904383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.904350 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw"] Nov 02 20:26:27.908771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.908743 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.959451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.959421 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1bbd80-openshift-default-dockercfg-cvlgq\"" Nov 02 20:26:27.994035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994001 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884d91e8-092f-4610-9d29-638a46aad29d-istiod-ca-cert\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994047 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884d91e8-092f-4610-9d29-638a46aad29d-istio-podinfo\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994068 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-certs\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994140 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-istio-token\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994299 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dsq5\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-kube-api-access-9dsq5\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994330 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-credential-socket\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994404 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-socket\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994449 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-data\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:27.994613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:27.994482 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-envoy\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.004237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.004187 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" podStartSLOduration=2.004171349 podStartE2EDuration="2.004171349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:27.939509588 +0000 UTC m=+2286.456282174" watchObservedRunningTime="2026-11-02 20:26:28.004171349 +0000 UTC m=+2286.520943933" Nov 02 20:26:28.007467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.007438 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw"] Nov 02 20:26:28.096109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.096013 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884d91e8-092f-4610-9d29-638a46aad29d-istiod-ca-cert\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.096109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.096071 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884d91e8-092f-4610-9d29-638a46aad29d-istio-podinfo\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.096109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.096093 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-certs\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.096424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.096234 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-istio-token\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.096991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.096959 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884d91e8-092f-4610-9d29-638a46aad29d-istiod-ca-cert\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.097252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.097224 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-certs\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.097389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.097336 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dsq5\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-kube-api-access-9dsq5\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.097480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.097387 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-credential-socket\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.097480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.097453 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-socket\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.097594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.097520 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-data\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.097594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.097561 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-envoy\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.098143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.098111 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-credential-socket\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.098373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.098350 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-socket\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.098502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.098406 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-data\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.100550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.100104 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-envoy\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.100550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.100513 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884d91e8-092f-4610-9d29-638a46aad29d-istio-podinfo\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.152999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.152935 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-istio-token\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.153163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.153001 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dsq5\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-kube-api-access-9dsq5\") pod \"e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.156131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.156082 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" podStartSLOduration=2.156066808 podStartE2EDuration="2.156066808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:28.154794798 +0000 UTC m=+2286.671567385" watchObservedRunningTime="2026-11-02 20:26:28.156066808 +0000 UTC m=+2286.672839391" Nov 02 20:26:28.223472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.223438 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:28.681972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.681943 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw"] Nov 02 20:26:28.682711 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:28.682687 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884d91e8_092f_4610_9d29_638a46aad29d.slice/crio-20a0bf5213712df5e169af68d36eae2d13e48608b3b8e060a30e54bfa89f78d1 WatchSource:0}: Error finding container 20a0bf5213712df5e169af68d36eae2d13e48608b3b8e060a30e54bfa89f78d1: Status 404 returned error can't find the container with id 20a0bf5213712df5e169af68d36eae2d13e48608b3b8e060a30e54bfa89f78d1 Nov 02 20:26:28.685396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.685362 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:28.685467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.685432 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:28.685510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.685463 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:28.856951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.856901 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" event={"ID":"884d91e8-092f-4610-9d29-638a46aad29d","Type":"ContainerStarted","Data":"942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25"} Nov 02 20:26:28.857487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.856962 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" event={"ID":"884d91e8-092f-4610-9d29-638a46aad29d","Type":"ContainerStarted","Data":"20a0bf5213712df5e169af68d36eae2d13e48608b3b8e060a30e54bfa89f78d1"} Nov 02 20:26:28.971039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:28.970924 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podStartSLOduration=1.9709084940000001 podStartE2EDuration="1.970908494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:28.968429189 +0000 UTC m=+2287.485201775" watchObservedRunningTime="2026-11-02 20:26:28.970908494 +0000 UTC m=+2287.487681078" Nov 02 20:26:29.223832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:29.223741 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:29.225316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:29.225257 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 20:26:29.225463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:29.225375 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 20:26:30.234913 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:30.234882 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:30.871146 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:30.871116 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:30.872357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:30.872335 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:26:35.023496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.023449 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:26:35.023896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.023541 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:26:35.639154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.639116 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c"] Nov 02 20:26:35.644568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.644541 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:35.695791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.695753 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c"] Nov 02 20:26:35.720434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.720365 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aaa18e-437d-4149-a284-e06b174ab477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:35.720613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.720515 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzrk\" (UniqueName: \"kubernetes.io/projected/25aaa18e-437d-4149-a284-e06b174ab477-kube-api-access-bmzrk\") pod \"payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:35.760307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.746253 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1bbd80\"" Nov 02 20:26:35.775655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.760705 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1bbd80-dockercfg-fjxfx\"" Nov 02 20:26:35.796272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.796236 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5"] Nov 02 20:26:35.800517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.800497 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:35.821616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.821590 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzrk\" (UniqueName: \"kubernetes.io/projected/25aaa18e-437d-4149-a284-e06b174ab477-kube-api-access-bmzrk\") pod \"payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:35.821799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.821779 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aaa18e-437d-4149-a284-e06b174ab477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:35.822519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.822500 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aaa18e-437d-4149-a284-e06b174ab477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:35.853378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.853337 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5"] Nov 02 20:26:35.878486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.878449 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzrk\" (UniqueName: \"kubernetes.io/projected/25aaa18e-437d-4149-a284-e06b174ab477-kube-api-access-bmzrk\") pod \"payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:35.923321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.923213 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c504e0-833e-41f5-899a-1fb20e1d2a50-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:35.923321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.923275 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt67n\" (UniqueName: \"kubernetes.io/projected/e3c504e0-833e-41f5-899a-1fb20e1d2a50-kube-api-access-gt67n\") pod \"payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:35.964412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:35.964367 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:36.024809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.024774 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt67n\" (UniqueName: \"kubernetes.io/projected/e3c504e0-833e-41f5-899a-1fb20e1d2a50-kube-api-access-gt67n\") pod \"payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:36.025453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.024997 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c504e0-833e-41f5-899a-1fb20e1d2a50-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:36.026806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.026032 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c504e0-833e-41f5-899a-1fb20e1d2a50-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:36.081992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.081949 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt67n\" (UniqueName: \"kubernetes.io/projected/e3c504e0-833e-41f5-899a-1fb20e1d2a50-kube-api-access-gt67n\") pod \"payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:36.115548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.115514 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:36.335248 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:36.335209 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25aaa18e_437d_4149_a284_e06b174ab477.slice/crio-dee35a7ea3feb1f39b702889759a782722af991814e4b0445874fcb42a1f1568 WatchSource:0}: Error finding container dee35a7ea3feb1f39b702889759a782722af991814e4b0445874fcb42a1f1568: Status 404 returned error can't find the container with id dee35a7ea3feb1f39b702889759a782722af991814e4b0445874fcb42a1f1568 Nov 02 20:26:36.335678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.335644 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c"] Nov 02 20:26:36.500663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.500633 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5"] Nov 02 20:26:36.501506 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:36.501481 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c504e0_833e_41f5_899a_1fb20e1d2a50.slice/crio-5f1ba6f97b9b34dc489420e0a856b88bd0c112cb63e73755e2b49aed5aa65300 WatchSource:0}: Error finding container 5f1ba6f97b9b34dc489420e0a856b88bd0c112cb63e73755e2b49aed5aa65300: Status 404 returned error can't find the container with id 5f1ba6f97b9b34dc489420e0a856b88bd0c112cb63e73755e2b49aed5aa65300 Nov 02 20:26:36.914943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.914906 5800 generic.go:356] "Generic (PLEG): container finished" podID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerID="353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c" exitCode=0 Nov 02 20:26:36.915110 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.914975 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" event={"ID":"a882e54b-94b4-4453-b990-4a9a64c7d434","Type":"ContainerDied","Data":"353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c"} Nov 02 20:26:36.915478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.915457 5800 scope.go:122] "RemoveContainer" containerID="353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c" Nov 02 20:26:36.916923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.916900 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" event={"ID":"25aaa18e-437d-4149-a284-e06b174ab477","Type":"ContainerStarted","Data":"b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024"} Nov 02 20:26:36.917063 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.916938 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" event={"ID":"25aaa18e-437d-4149-a284-e06b174ab477","Type":"ContainerStarted","Data":"dee35a7ea3feb1f39b702889759a782722af991814e4b0445874fcb42a1f1568"} Nov 02 20:26:36.917063 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.916961 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:36.917179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.917106 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:36.917243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.917194 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:36.917420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.917242 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:36.919159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.919134 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" event={"ID":"e3c504e0-833e-41f5-899a-1fb20e1d2a50","Type":"ContainerStarted","Data":"fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380"} Nov 02 20:26:36.919247 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.919177 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" event={"ID":"e3c504e0-833e-41f5-899a-1fb20e1d2a50","Type":"ContainerStarted","Data":"5f1ba6f97b9b34dc489420e0a856b88bd0c112cb63e73755e2b49aed5aa65300"} Nov 02 20:26:36.919328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:36.919318 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:37.131445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:37.131393 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" podStartSLOduration=2.131380993 podStartE2EDuration="2.131380993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:37.128726708 +0000 UTC m=+2295.645499294" watchObservedRunningTime="2026-11-02 20:26:37.131380993 +0000 UTC m=+2295.648153577" Nov 02 20:26:37.238665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:37.237706 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" podStartSLOduration=2.237686679 podStartE2EDuration="2.237686679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:37.234836635 +0000 UTC m=+2295.751609222" watchObservedRunningTime="2026-11-02 20:26:37.237686679 +0000 UTC m=+2295.754459271" Nov 02 20:26:37.953076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:37.953042 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" event={"ID":"a882e54b-94b4-4453-b990-4a9a64c7d434","Type":"ContainerStarted","Data":"9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17"} Nov 02 20:26:38.420985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:38.420947 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:26:38.421422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:38.421162 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 20:26:38.421422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:38.421223 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 20:26:38.859675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:38.859637 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:26:38.860125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:38.860101 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:26:39.421019 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:39.420983 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 20:26:39.421472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:39.421036 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 20:26:40.421701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:40.421664 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 20:26:40.422101 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:40.421718 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 20:26:41.421250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:41.421210 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 20:26:41.421446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:41.421266 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 20:26:42.421832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:42.421796 5800 patch_prober.go:29] interesting pod/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Nov 02 20:26:42.422240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:42.421853 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Nov 02 20:26:43.426188 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:43.426154 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:26:43.426730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:43.426629 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:26:43.427361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:43.427340 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:26:44.730483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.730445 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml"] Nov 02 20:26:44.734751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.734728 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.741842 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.741819 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-certs\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.741976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.741850 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-podinfo\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.741976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.741881 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-data\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.742124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.742014 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-socket\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.742124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.742074 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96c9\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-kube-api-access-l96c9\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.742244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.742150 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-token\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.742325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.742306 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-envoy\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.742400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.742383 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-credential-socket\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.742483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.742469 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istiod-ca-cert\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.811528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.811488 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cdbffd-openshift-default-dockercfg-h2vj6\"" Nov 02 20:26:44.835127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.835092 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml"] Nov 02 20:26:44.849791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.849750 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-credential-socket\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.849868 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istiod-ca-cert\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.849922 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-certs\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.849956 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-podinfo\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.849989 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-data\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.850021 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-socket\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.850076 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96c9\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-kube-api-access-l96c9\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.850114 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-token\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.850227 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-envoy\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.850233 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-credential-socket\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.850535 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-data\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.850696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.850674 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-socket\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.851211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.851181 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-certs\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.851388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.851232 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istiod-ca-cert\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.853318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.853272 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-envoy\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.854162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.854138 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-podinfo\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.924909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.924840 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96c9\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-kube-api-access-l96c9\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:44.925968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:44.925882 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-token\") pod \"e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:45.051686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:45.051648 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:45.472420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:45.472383 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml"] Nov 02 20:26:45.473185 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:45.473158 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ceafdd2_5bea_4a23_a8a5_89c34ddeaa15.slice/crio-cf8ec39f0848dccc72d598a5349e938664a618b45d3e5b085a524b04062b0c8d WatchSource:0}: Error finding container cf8ec39f0848dccc72d598a5349e938664a618b45d3e5b085a524b04062b0c8d: Status 404 returned error can't find the container with id cf8ec39f0848dccc72d598a5349e938664a618b45d3e5b085a524b04062b0c8d Nov 02 20:26:45.475623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:45.475590 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:45.475697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:45.475659 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:45.475741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:45.475694 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:26:46.014051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:46.014018 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" event={"ID":"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15","Type":"ContainerStarted","Data":"4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2"} Nov 02 20:26:46.014500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:46.014074 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" event={"ID":"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15","Type":"ContainerStarted","Data":"cf8ec39f0848dccc72d598a5349e938664a618b45d3e5b085a524b04062b0c8d"} Nov 02 20:26:46.052250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:46.052209 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:46.059020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:46.058989 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:46.125255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:46.125206 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podStartSLOduration=2.125192431 podStartE2EDuration="2.125192431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:46.122107421 +0000 UTC m=+2304.638880006" watchObservedRunningTime="2026-11-02 20:26:46.125192431 +0000 UTC m=+2304.641965085" Nov 02 20:26:47.021855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:47.021815 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:47.022894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:47.022873 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:26:47.956128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:47.956097 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:26:47.956353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:47.956163 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:26:52.487093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.487053 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs"] Nov 02 20:26:52.492514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.492492 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.551036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.550992 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs"] Nov 02 20:26:52.552161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.552029 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f9487c-af13-49ba-a9b2-82963885abdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.552480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.552462 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxswz\" (UniqueName: \"kubernetes.io/projected/71f9487c-af13-49ba-a9b2-82963885abdf-kube-api-access-sxswz\") pod \"payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.587782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.587747 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cdbffd\"" Nov 02 20:26:52.588142 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.588091 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cdbffd-dockercfg-vkk5w\"" Nov 02 20:26:52.649983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.649947 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz"] Nov 02 20:26:52.653733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.653703 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f9487c-af13-49ba-a9b2-82963885abdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.653879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.653834 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxswz\" (UniqueName: \"kubernetes.io/projected/71f9487c-af13-49ba-a9b2-82963885abdf-kube-api-access-sxswz\") pod \"payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.654271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.654253 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:52.654711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.654690 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f9487c-af13-49ba-a9b2-82963885abdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.697646 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:26:52.697595 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:26:52.707997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.707970 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxswz\" (UniqueName: \"kubernetes.io/projected/71f9487c-af13-49ba-a9b2-82963885abdf-kube-api-access-sxswz\") pod \"payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.708156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.708116 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz"] Nov 02 20:26:52.755269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.755236 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d493684-7763-4db0-83f2-3b4726401899-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:52.755491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.755395 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5hs\" (UniqueName: \"kubernetes.io/projected/0d493684-7763-4db0-83f2-3b4726401899-kube-api-access-kc5hs\") pod \"payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:52.807685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.807653 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:52.857929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.856515 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5hs\" (UniqueName: \"kubernetes.io/projected/0d493684-7763-4db0-83f2-3b4726401899-kube-api-access-kc5hs\") pod \"payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:52.857929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.856710 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d493684-7763-4db0-83f2-3b4726401899-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:52.858652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.858626 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d493684-7763-4db0-83f2-3b4726401899-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:52.913536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.913493 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5hs\" (UniqueName: \"kubernetes.io/projected/0d493684-7763-4db0-83f2-3b4726401899-kube-api-access-kc5hs\") pod \"payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:52.970703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:52.970662 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:53.192542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:53.192506 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs"] Nov 02 20:26:53.192950 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:53.192921 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f9487c_af13_49ba_a9b2_82963885abdf.slice/crio-6935bb6ab5c65f6c76fe55e3ccb9f85120ba2563fa142d26dc73a7e131915049 WatchSource:0}: Error finding container 6935bb6ab5c65f6c76fe55e3ccb9f85120ba2563fa142d26dc73a7e131915049: Status 404 returned error can't find the container with id 6935bb6ab5c65f6c76fe55e3ccb9f85120ba2563fa142d26dc73a7e131915049 Nov 02 20:26:53.357268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:53.357237 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz"] Nov 02 20:26:53.358627 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:26:53.358603 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d493684_7763_4db0_83f2_3b4726401899.slice/crio-dae67ededae40fa6266c42996c551cf0e1b0066ca485b1050ba750a6200875a3 WatchSource:0}: Error finding container dae67ededae40fa6266c42996c551cf0e1b0066ca485b1050ba750a6200875a3: Status 404 returned error can't find the container with id dae67ededae40fa6266c42996c551cf0e1b0066ca485b1050ba750a6200875a3 Nov 02 20:26:54.079694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.079662 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" event={"ID":"0d493684-7763-4db0-83f2-3b4726401899","Type":"ContainerStarted","Data":"57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b"} Nov 02 20:26:54.080105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.079719 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" event={"ID":"0d493684-7763-4db0-83f2-3b4726401899","Type":"ContainerStarted","Data":"dae67ededae40fa6266c42996c551cf0e1b0066ca485b1050ba750a6200875a3"} Nov 02 20:26:54.080105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.079754 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:26:54.081327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.081261 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" event={"ID":"71f9487c-af13-49ba-a9b2-82963885abdf","Type":"ContainerStarted","Data":"f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722"} Nov 02 20:26:54.081452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.081328 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" event={"ID":"71f9487c-af13-49ba-a9b2-82963885abdf","Type":"ContainerStarted","Data":"6935bb6ab5c65f6c76fe55e3ccb9f85120ba2563fa142d26dc73a7e131915049"} Nov 02 20:26:54.081452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.081349 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:26:54.185265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.185209 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" podStartSLOduration=2.185191546 podStartE2EDuration="2.185191546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:54.184512631 +0000 UTC m=+2312.701285217" watchObservedRunningTime="2026-11-02 20:26:54.185191546 +0000 UTC m=+2312.701964132" Nov 02 20:26:54.301306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:26:54.293955 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" podStartSLOduration=2.293936669 podStartE2EDuration="2.293936669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:54.289951969 +0000 UTC m=+2312.806724554" watchObservedRunningTime="2026-11-02 20:26:54.293936669 +0000 UTC m=+2312.810709255" Nov 02 20:27:05.090895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:05.090861 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:27:05.091423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:05.090938 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:27:20.835169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:20.835118 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm"] Nov 02 20:27:20.935703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:20.935663 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr"] Nov 02 20:27:22.698802 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:27:22.698762 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:27:25.835963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:25.835918 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" podUID="140a2cc6-38b0-4eed-ae10-07be0897848e" containerName="payload-processing" containerID="cri-o://8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96" gracePeriod=25 Nov 02 20:27:25.936391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:25.936329 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" podUID="cd836709-a27c-4734-93ba-6bd87fd7f9f8" containerName="payload-pre-processing" containerID="cri-o://7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8" gracePeriod=25 Nov 02 20:27:26.137524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.137492 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:27:26.235073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.235047 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:27:26.273428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.273398 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/140a2cc6-38b0-4eed-ae10-07be0897848e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140a2cc6-38b0-4eed-ae10-07be0897848e-plugins-config-volume\") pod \"140a2cc6-38b0-4eed-ae10-07be0897848e\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " Nov 02 20:27:26.273631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.273583 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140a2cc6-38b0-4eed-ae10-07be0897848e-kube-api-access-h76hg\" (UniqueName: \"kubernetes.io/projected/140a2cc6-38b0-4eed-ae10-07be0897848e-kube-api-access-h76hg\") pod \"140a2cc6-38b0-4eed-ae10-07be0897848e\" (UID: \"140a2cc6-38b0-4eed-ae10-07be0897848e\") " Nov 02 20:27:26.273897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.273866 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140a2cc6-38b0-4eed-ae10-07be0897848e-plugins-config-volume" pod "140a2cc6-38b0-4eed-ae10-07be0897848e" (UID: "140a2cc6-38b0-4eed-ae10-07be0897848e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:27:26.274207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.274182 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140a2cc6-38b0-4eed-ae10-07be0897848e-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:26.275979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.275957 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140a2cc6-38b0-4eed-ae10-07be0897848e-kube-api-access-h76hg" pod "140a2cc6-38b0-4eed-ae10-07be0897848e" (UID: "140a2cc6-38b0-4eed-ae10-07be0897848e"). InnerVolumeSpecName "kube-api-access-h76hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:27:26.322053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.322020 5800 generic.go:356] "Generic (PLEG): container finished" podID="cd836709-a27c-4734-93ba-6bd87fd7f9f8" containerID="7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8" exitCode=0 Nov 02 20:27:26.322225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.322085 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" Nov 02 20:27:26.322225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.322105 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" event={"ID":"cd836709-a27c-4734-93ba-6bd87fd7f9f8","Type":"ContainerDied","Data":"7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8"} Nov 02 20:27:26.322225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.322160 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr" event={"ID":"cd836709-a27c-4734-93ba-6bd87fd7f9f8","Type":"ContainerDied","Data":"f19ec4a75c5193fc11fcd71910f1ab01add480c8265e96c9d5f13a90a569abf2"} Nov 02 20:27:26.322225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.322182 5800 scope.go:122] "RemoveContainer" containerID="7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8" Nov 02 20:27:26.323689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.323662 5800 generic.go:356] "Generic (PLEG): container finished" podID="140a2cc6-38b0-4eed-ae10-07be0897848e" containerID="8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96" exitCode=0 Nov 02 20:27:26.323800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.323708 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" Nov 02 20:27:26.323800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.323761 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" event={"ID":"140a2cc6-38b0-4eed-ae10-07be0897848e","Type":"ContainerDied","Data":"8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96"} Nov 02 20:27:26.323891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.323802 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm" event={"ID":"140a2cc6-38b0-4eed-ae10-07be0897848e","Type":"ContainerDied","Data":"e0640021149ceac28ab4e41667e2c7a24c7209248c495d7ef51f5eafc4ffc53a"} Nov 02 20:27:26.333053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.333030 5800 scope.go:122] "RemoveContainer" containerID="7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8" Nov 02 20:27:26.333382 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:27:26.333360 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8\": container with ID starting with 7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8 not found: ID does not exist" containerID="7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8" Nov 02 20:27:26.333480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.333388 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8"} err="failed to get container status \"7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8\": rpc error: code = NotFound desc = could not find container \"7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8\": container with ID starting with 7662dee9684f6e5b67ef70b43244592060d190ab4df4b6b88086ca3dab769de8 not found: ID does not exist" Nov 02 20:27:26.333480 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.333406 5800 scope.go:122] "RemoveContainer" containerID="8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96" Nov 02 20:27:26.343587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.343562 5800 scope.go:122] "RemoveContainer" containerID="8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96" Nov 02 20:27:26.343891 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:27:26.343872 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96\": container with ID starting with 8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96 not found: ID does not exist" containerID="8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96" Nov 02 20:27:26.343973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.343898 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96"} err="failed to get container status \"8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96\": rpc error: code = NotFound desc = could not find container \"8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96\": container with ID starting with 8ce9d7ddee453b1f45a7aebcf9038a5447c3da70f20f40d1a6310392269cfe96 not found: ID does not exist" Nov 02 20:27:26.375086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.374997 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd836709-a27c-4734-93ba-6bd87fd7f9f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd836709-a27c-4734-93ba-6bd87fd7f9f8-plugins-config-volume\") pod \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " Nov 02 20:27:26.375086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.375041 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd836709-a27c-4734-93ba-6bd87fd7f9f8-kube-api-access-4bvrq\" (UniqueName: \"kubernetes.io/projected/cd836709-a27c-4734-93ba-6bd87fd7f9f8-kube-api-access-4bvrq\") pod \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\" (UID: \"cd836709-a27c-4734-93ba-6bd87fd7f9f8\") " Nov 02 20:27:26.375772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.375744 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd836709-a27c-4734-93ba-6bd87fd7f9f8-plugins-config-volume" pod "cd836709-a27c-4734-93ba-6bd87fd7f9f8" (UID: "cd836709-a27c-4734-93ba-6bd87fd7f9f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:27:26.375980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.375962 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h76hg\" (UniqueName: \"kubernetes.io/projected/140a2cc6-38b0-4eed-ae10-07be0897848e-kube-api-access-h76hg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:26.376055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.375985 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd836709-a27c-4734-93ba-6bd87fd7f9f8-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:26.380058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.380035 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd836709-a27c-4734-93ba-6bd87fd7f9f8-kube-api-access-4bvrq" pod "cd836709-a27c-4734-93ba-6bd87fd7f9f8" (UID: "cd836709-a27c-4734-93ba-6bd87fd7f9f8"). InnerVolumeSpecName "kube-api-access-4bvrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:27:26.477628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.477588 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bvrq\" (UniqueName: \"kubernetes.io/projected/cd836709-a27c-4734-93ba-6bd87fd7f9f8-kube-api-access-4bvrq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:27:26.517069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.517032 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm"] Nov 02 20:27:26.521325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.521298 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6f1fd4-7df59dfc57-vdsnm"] Nov 02 20:27:26.737814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.737779 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr"] Nov 02 20:27:26.742299 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:26.742254 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6f1fd4-76cb7b4dd7-689zr"] Nov 02 20:27:27.623042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:27.623010 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140a2cc6-38b0-4eed-ae10-07be0897848e" path="/var/lib/kubelet/pods/140a2cc6-38b0-4eed-ae10-07be0897848e/volumes" Nov 02 20:27:27.623471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:27.623458 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd836709-a27c-4734-93ba-6bd87fd7f9f8" path="/var/lib/kubelet/pods/cd836709-a27c-4734-93ba-6bd87fd7f9f8/volumes" Nov 02 20:27:30.334127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:30.334039 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r_740e2f52-591a-4712-8898-a14aa59dd2a9/payload-processing/0.log" Nov 02 20:27:30.546782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:30.546752 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r_740e2f52-591a-4712-8898-a14aa59dd2a9/payload-processing/0.log" Nov 02 20:27:52.698160 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:27:52.698119 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:27:59.576725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:59.576690 5800 generic.go:356] "Generic (PLEG): container finished" podID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerID="dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9" exitCode=0 Nov 02 20:27:59.577317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:59.576767 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" event={"ID":"51b0d508-997e-4a26-9fa6-f02b398ac7ce","Type":"ContainerDied","Data":"dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9"} Nov 02 20:27:59.577317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:59.577221 5800 scope.go:122] "RemoveContainer" containerID="dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9" Nov 02 20:27:59.578826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:59.578792 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:59.578944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:59.578883 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:59.578944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:27:59.578934 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:00.585885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:00.585850 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" event={"ID":"51b0d508-997e-4a26-9fa6-f02b398ac7ce","Type":"ContainerStarted","Data":"8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6"} Nov 02 20:28:01.042307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:01.041684 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:28:01.050441 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:01.050398 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 20:28:01.050643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:01.050474 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 20:28:02.050428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:02.050386 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 20:28:02.051048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:02.051014 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 20:28:03.042740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:03.042697 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 20:28:03.043073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:03.043042 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 20:28:03.745234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:03.744322 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:28:04.045695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:04.045659 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 20:28:04.046035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:04.046006 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 20:28:05.046304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:05.040679 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 02 20:28:05.046304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:05.040754 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 02 20:28:06.045220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:06.045181 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:28:06.046294 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:06.046255 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:28:12.735243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.735204 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc"] Nov 02 20:28:12.735914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.735893 5800 cpu_manager.go:414] "containerMap: removing container" podUID="140a2cc6-38b0-4eed-ae10-07be0897848e" containerName="payload-processing" Nov 02 20:28:12.735998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.735914 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140a2cc6-38b0-4eed-ae10-07be0897848e" containerName="payload-processing" Nov 02 20:28:12.735998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.735930 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cd836709-a27c-4734-93ba-6bd87fd7f9f8" containerName="payload-pre-processing" Nov 02 20:28:12.735998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.735936 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd836709-a27c-4734-93ba-6bd87fd7f9f8" containerName="payload-pre-processing" Nov 02 20:28:12.736164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.736029 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="140a2cc6-38b0-4eed-ae10-07be0897848e" containerName="payload-processing" Nov 02 20:28:12.736164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.736037 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd836709-a27c-4734-93ba-6bd87fd7f9f8" containerName="payload-pre-processing" Nov 02 20:28:12.739883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.739857 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.835399 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.835366 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:28:12.835674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.835408 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:28:12.835674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.835560 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:28:12.835827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.835616 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:28:12.840653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.840626 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc"] Nov 02 20:28:12.881662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.881627 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.881873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.881716 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.881873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.881787 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/2f07eaaa-53cc-4410-bd6e-b8431f435d10-kube-api-access-hphh5\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.982598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.982562 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.982802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.982615 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.982802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.982642 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/2f07eaaa-53cc-4410-bd6e-b8431f435d10-kube-api-access-hphh5\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.983410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.983384 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:12.985111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:12.985082 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:13.037143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:13.037110 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/2f07eaaa-53cc-4410-bd6e-b8431f435d10-kube-api-access-hphh5\") pod \"maas-api-revoke-sub-78e0174a-4v4xc\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:13.059236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:13.059203 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:13.440264 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:28:13.440225 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f07eaaa_53cc_4410_bd6e_b8431f435d10.slice/crio-cbf106f188bd42f49676184eb29548dd82ab1d04031e3df5a0ac8596886c9777 WatchSource:0}: Error finding container cbf106f188bd42f49676184eb29548dd82ab1d04031e3df5a0ac8596886c9777: Status 404 returned error can't find the container with id cbf106f188bd42f49676184eb29548dd82ab1d04031e3df5a0ac8596886c9777 Nov 02 20:28:13.443197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:13.443127 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc"] Nov 02 20:28:13.690602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:13.690519 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" event={"ID":"2f07eaaa-53cc-4410-bd6e-b8431f435d10","Type":"ContainerStarted","Data":"cbf106f188bd42f49676184eb29548dd82ab1d04031e3df5a0ac8596886c9777"} Nov 02 20:28:14.263368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.263324 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf"] Nov 02 20:28:14.263889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.263576 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" podUID="4f04350c-cf11-4297-bcb0-9bc733c19a28" containerName="payload-processing" containerID="cri-o://10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567" gracePeriod=30 Nov 02 20:28:14.330386 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.330345 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh"] Nov 02 20:28:14.330680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.330651 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" podUID="173083ea-5a01-4fb6-94ff-7c63d8a6bd10" containerName="payload-pre-processing" containerID="cri-o://dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e" gracePeriod=30 Nov 02 20:28:14.610199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.610147 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:28:14.640162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.640125 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:28:14.699172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.699139 5800 generic.go:356] "Generic (PLEG): container finished" podID="2f07eaaa-53cc-4410-bd6e-b8431f435d10" containerID="1b0e96cf8f5e2acc1b0db7f4d8cadf54af8cb7df6b0f2da58a84527652e0c84c" exitCode=0 Nov 02 20:28:14.699385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.699187 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" event={"ID":"2f07eaaa-53cc-4410-bd6e-b8431f435d10","Type":"ContainerDied","Data":"1b0e96cf8f5e2acc1b0db7f4d8cadf54af8cb7df6b0f2da58a84527652e0c84c"} Nov 02 20:28:14.700568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.700545 5800 generic.go:356] "Generic (PLEG): container finished" podID="173083ea-5a01-4fb6-94ff-7c63d8a6bd10" containerID="dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e" exitCode=0 Nov 02 20:28:14.700685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.700578 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" Nov 02 20:28:14.700685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.700628 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" event={"ID":"173083ea-5a01-4fb6-94ff-7c63d8a6bd10","Type":"ContainerDied","Data":"dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e"} Nov 02 20:28:14.700812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.700686 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh" event={"ID":"173083ea-5a01-4fb6-94ff-7c63d8a6bd10","Type":"ContainerDied","Data":"4d0483aecf40682a5314b24418a95ae614b97e984b97a62f08630aa674c2696e"} Nov 02 20:28:14.700812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.700721 5800 scope.go:122] "RemoveContainer" containerID="dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e" Nov 02 20:28:14.701968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.701941 5800 generic.go:356] "Generic (PLEG): container finished" podID="4f04350c-cf11-4297-bcb0-9bc733c19a28" containerID="10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567" exitCode=0 Nov 02 20:28:14.702097 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.701997 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" event={"ID":"4f04350c-cf11-4297-bcb0-9bc733c19a28","Type":"ContainerDied","Data":"10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567"} Nov 02 20:28:14.702097 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.702026 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" event={"ID":"4f04350c-cf11-4297-bcb0-9bc733c19a28","Type":"ContainerDied","Data":"5803196dd78f773946216ccc781606dfcd688e3754a85e6c96ea2b8a811a0818"} Nov 02 20:28:14.702097 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.702037 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf" Nov 02 20:28:14.703221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.703201 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-kube-api-access-fx4qv\" (UniqueName: \"kubernetes.io/projected/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-kube-api-access-fx4qv\") pod \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " Nov 02 20:28:14.703419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.703399 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f04350c-cf11-4297-bcb0-9bc733c19a28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f04350c-cf11-4297-bcb0-9bc733c19a28-plugins-config-volume\") pod \"4f04350c-cf11-4297-bcb0-9bc733c19a28\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " Nov 02 20:28:14.703494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.703466 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-plugins-config-volume\") pod \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\" (UID: \"173083ea-5a01-4fb6-94ff-7c63d8a6bd10\") " Nov 02 20:28:14.703560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.703544 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f04350c-cf11-4297-bcb0-9bc733c19a28-kube-api-access-jcd4j\" (UniqueName: \"kubernetes.io/projected/4f04350c-cf11-4297-bcb0-9bc733c19a28-kube-api-access-jcd4j\") pod \"4f04350c-cf11-4297-bcb0-9bc733c19a28\" (UID: \"4f04350c-cf11-4297-bcb0-9bc733c19a28\") " Nov 02 20:28:14.703919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.703840 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f04350c-cf11-4297-bcb0-9bc733c19a28-plugins-config-volume" pod "4f04350c-cf11-4297-bcb0-9bc733c19a28" (UID: "4f04350c-cf11-4297-bcb0-9bc733c19a28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:14.703919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.703855 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-plugins-config-volume" pod "173083ea-5a01-4fb6-94ff-7c63d8a6bd10" (UID: "173083ea-5a01-4fb6-94ff-7c63d8a6bd10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:14.704416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.704389 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f04350c-cf11-4297-bcb0-9bc733c19a28-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.704514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.704420 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.705686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.705663 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-kube-api-access-fx4qv" pod "173083ea-5a01-4fb6-94ff-7c63d8a6bd10" (UID: "173083ea-5a01-4fb6-94ff-7c63d8a6bd10"). InnerVolumeSpecName "kube-api-access-fx4qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:14.706243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.706220 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f04350c-cf11-4297-bcb0-9bc733c19a28-kube-api-access-jcd4j" pod "4f04350c-cf11-4297-bcb0-9bc733c19a28" (UID: "4f04350c-cf11-4297-bcb0-9bc733c19a28"). InnerVolumeSpecName "kube-api-access-jcd4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:14.714474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.714448 5800 scope.go:122] "RemoveContainer" containerID="dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e" Nov 02 20:28:14.714836 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:14.714808 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e\": container with ID starting with dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e not found: ID does not exist" containerID="dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e" Nov 02 20:28:14.714962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.714843 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e"} err="failed to get container status \"dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e\": rpc error: code = NotFound desc = could not find container \"dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e\": container with ID starting with dfef2098802bbe7b3bbe9bb5bef820e4d85c99af96fce5216113201278eb696e not found: ID does not exist" Nov 02 20:28:14.714962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.714859 5800 scope.go:122] "RemoveContainer" containerID="10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567" Nov 02 20:28:14.727028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.727009 5800 scope.go:122] "RemoveContainer" containerID="10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567" Nov 02 20:28:14.727358 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:14.727329 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567\": container with ID starting with 10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567 not found: ID does not exist" containerID="10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567" Nov 02 20:28:14.727450 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.727366 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567"} err="failed to get container status \"10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567\": rpc error: code = NotFound desc = could not find container \"10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567\": container with ID starting with 10e77176c58df2edfe668ded509182c8d510f53c493d00acb42fa348967e2567 not found: ID does not exist" Nov 02 20:28:14.814382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.806555 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcd4j\" (UniqueName: \"kubernetes.io/projected/4f04350c-cf11-4297-bcb0-9bc733c19a28-kube-api-access-jcd4j\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:14.814382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:14.806599 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4qv\" (UniqueName: \"kubernetes.io/projected/173083ea-5a01-4fb6-94ff-7c63d8a6bd10-kube-api-access-fx4qv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:15.117989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.117871 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh"] Nov 02 20:28:15.121425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.121396 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-57e2cf-7f67c6c84c-sxqsh"] Nov 02 20:28:15.222456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.222422 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf"] Nov 02 20:28:15.228122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.228095 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-57e2cf-bf7b57cdb-dqftf"] Nov 02 20:28:15.624169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.624127 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="173083ea-5a01-4fb6-94ff-7c63d8a6bd10" path="/var/lib/kubelet/pods/173083ea-5a01-4fb6-94ff-7c63d8a6bd10/volumes" Nov 02 20:28:15.624748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.624724 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f04350c-cf11-4297-bcb0-9bc733c19a28" path="/var/lib/kubelet/pods/4f04350c-cf11-4297-bcb0-9bc733c19a28/volumes" Nov 02 20:28:15.877941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.877879 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:15.919837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.919808 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f07eaaa-53cc-4410-bd6e-b8431f435d10-kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/2f07eaaa-53cc-4410-bd6e-b8431f435d10-kube-api-access-hphh5\") pod \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " Nov 02 20:28:15.919993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.919980 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-xks-ca\") pod \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " Nov 02 20:28:15.920098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.920003 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-openshift-ca\") pod \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\" (UID: \"2f07eaaa-53cc-4410-bd6e-b8431f435d10\") " Nov 02 20:28:15.920484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.920455 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-openshift-ca" pod "2f07eaaa-53cc-4410-bd6e-b8431f435d10" (UID: "2f07eaaa-53cc-4410-bd6e-b8431f435d10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:15.921835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.921811 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-xks-ca" pod "2f07eaaa-53cc-4410-bd6e-b8431f435d10" (UID: "2f07eaaa-53cc-4410-bd6e-b8431f435d10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:28:15.922041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:15.922018 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f07eaaa-53cc-4410-bd6e-b8431f435d10-kube-api-access-hphh5" pod "2f07eaaa-53cc-4410-bd6e-b8431f435d10" (UID: "2f07eaaa-53cc-4410-bd6e-b8431f435d10"). InnerVolumeSpecName "kube-api-access-hphh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:16.021705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:16.021665 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:16.021705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:16.021707 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f07eaaa-53cc-4410-bd6e-b8431f435d10-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:16.021912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:16.021725 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hphh5\" (UniqueName: \"kubernetes.io/projected/2f07eaaa-53cc-4410-bd6e-b8431f435d10-kube-api-access-hphh5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:16.719310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:16.719259 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" event={"ID":"2f07eaaa-53cc-4410-bd6e-b8431f435d10","Type":"ContainerDied","Data":"cbf106f188bd42f49676184eb29548dd82ab1d04031e3df5a0ac8596886c9777"} Nov 02 20:28:16.719310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:16.719300 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc" Nov 02 20:28:16.719715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:16.719332 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf106f188bd42f49676184eb29548dd82ab1d04031e3df5a0ac8596886c9777" Nov 02 20:28:20.237269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:20.237226 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g"] Nov 02 20:28:20.237784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:20.237527 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" containerID="cri-o://5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd" gracePeriod=30 Nov 02 20:28:22.081561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:22.081527 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:28:22.084866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:22.084840 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:28:22.698328 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:22.698261 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:28:25.565463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.565434 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:28:25.659947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.659914 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-token\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-token\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.660217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660193 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-certs\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.660377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660250 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-credential-socket\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.660377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660359 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-socket\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.660477 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660462 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-envoy\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.660548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660532 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-kube-api-access-mbxcb\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-kube-api-access-mbxcb\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.660606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660567 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-data\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.660677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660661 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-podinfo\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.662018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660794 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-socket" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:25.662143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660953 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-credential-socket" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:25.662143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.660990 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-data" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:25.662143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.661010 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-certs" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:25.662143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662100 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1173c959-8c68-4379-9ade-818f7d7e8fee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1173c959-8c68-4379-9ade-818f7d7e8fee-istiod-ca-cert\") pod \"1173c959-8c68-4379-9ade-818f7d7e8fee\" (UID: \"1173c959-8c68-4379-9ade-818f7d7e8fee\") " Nov 02 20:28:25.662879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662574 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1173c959-8c68-4379-9ade-818f7d7e8fee-istiod-ca-cert" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:25.662879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662720 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.662879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662740 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.662879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662743 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-token" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:25.662879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662753 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1173c959-8c68-4379-9ade-818f7d7e8fee-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.662879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662800 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.662879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.662816 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.663300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.663257 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-kube-api-access-mbxcb" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "kube-api-access-mbxcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:25.669663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.669636 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-podinfo" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:28:25.669936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.669916 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-envoy" pod "1173c959-8c68-4379-9ade-818f7d7e8fee" (UID: "1173c959-8c68-4379-9ade-818f7d7e8fee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:25.763953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.763847 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.763953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.763888 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.763953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.763908 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxcb\" (UniqueName: \"kubernetes.io/projected/1173c959-8c68-4379-9ade-818f7d7e8fee-kube-api-access-mbxcb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.763953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.763923 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1173c959-8c68-4379-9ade-818f7d7e8fee-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:25.798445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.798413 5800 generic.go:356] "Generic (PLEG): container finished" podID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerID="5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd" exitCode=0 Nov 02 20:28:25.798635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.798470 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" Nov 02 20:28:25.798635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.798479 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" event={"ID":"1173c959-8c68-4379-9ade-818f7d7e8fee","Type":"ContainerDied","Data":"5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd"} Nov 02 20:28:25.798635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.798546 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g" event={"ID":"1173c959-8c68-4379-9ade-818f7d7e8fee","Type":"ContainerDied","Data":"4a3041c4e8f907d5b64590c6ac93de35c45683f4d370e9da540d0c1e2aef557c"} Nov 02 20:28:25.798635 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.798575 5800 scope.go:122] "RemoveContainer" containerID="5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd" Nov 02 20:28:25.809739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.809722 5800 scope.go:122] "RemoveContainer" containerID="32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471" Nov 02 20:28:25.819455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.819438 5800 scope.go:122] "RemoveContainer" containerID="5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd" Nov 02 20:28:25.819698 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:25.819680 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd\": container with ID starting with 5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd not found: ID does not exist" containerID="5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd" Nov 02 20:28:25.819775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.819710 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd"} err="failed to get container status \"5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd\": rpc error: code = NotFound desc = could not find container \"5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd\": container with ID starting with 5006ac02eee3d8b4c5aec7a06733ce2a44f9951e2a2e6b4e68c2403c0cac48fd not found: ID does not exist" Nov 02 20:28:25.819775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.819741 5800 scope.go:122] "RemoveContainer" containerID="32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471" Nov 02 20:28:25.819990 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:25.819972 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471\": container with ID starting with 32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471 not found: ID does not exist" containerID="32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471" Nov 02 20:28:25.820032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.819996 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471"} err="failed to get container status \"32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471\": rpc error: code = NotFound desc = could not find container \"32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471\": container with ID starting with 32925851eb768b7168edd229882f04f0700e467b7b6917db73b7658cd5764471 not found: ID does not exist" Nov 02 20:28:25.955126 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.955092 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g"] Nov 02 20:28:25.961239 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:25.961207 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-57e2cf-openshift-default-6b567447c9-x9x5g"] Nov 02 20:28:27.624089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:27.624053 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" path="/var/lib/kubelet/pods/1173c959-8c68-4379-9ade-818f7d7e8fee/volumes" Nov 02 20:28:29.829932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:29.829898 5800 generic.go:356] "Generic (PLEG): container finished" podID="884d91e8-092f-4610-9d29-638a46aad29d" containerID="942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25" exitCode=0 Nov 02 20:28:29.830448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:29.829970 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" event={"ID":"884d91e8-092f-4610-9d29-638a46aad29d","Type":"ContainerDied","Data":"942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25"} Nov 02 20:28:29.830448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:29.830433 5800 scope.go:122] "RemoveContainer" containerID="942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25" Nov 02 20:28:29.831958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:29.831932 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:29.832036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:29.832001 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:29.832075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:29.832033 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:30.855617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:30.855578 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" event={"ID":"884d91e8-092f-4610-9d29-638a46aad29d","Type":"ContainerStarted","Data":"74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8"} Nov 02 20:28:30.871641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:30.871611 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:28:31.223741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:31.223651 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:28:31.224711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:31.224682 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 20:28:31.224828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:31.224746 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 20:28:32.224263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:32.224225 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 20:28:32.224666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:32.224317 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 20:28:33.224438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:33.224277 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 20:28:33.224438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:33.224359 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 20:28:34.224220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:34.224184 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 20:28:34.224440 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:34.224268 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 20:28:34.271955 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:34.271921 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z"] Nov 02 20:28:34.274132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:34.274095 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" containerID="cri-o://9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17" gracePeriod=30 Nov 02 20:28:35.223961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:35.223928 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Nov 02 20:28:35.224203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:35.223990 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Nov 02 20:28:36.229168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.229127 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:28:36.230240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.230214 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.366136 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw"] Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367066 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367087 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367108 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367116 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367132 5800 cpu_manager.go:414] "containerMap: removing container" podUID="173083ea-5a01-4fb6-94ff-7c63d8a6bd10" containerName="payload-pre-processing" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367140 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173083ea-5a01-4fb6-94ff-7c63d8a6bd10" containerName="payload-pre-processing" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367181 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2f07eaaa-53cc-4410-bd6e-b8431f435d10" containerName="revoke-keys" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367189 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f07eaaa-53cc-4410-bd6e-b8431f435d10" containerName="revoke-keys" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367208 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4f04350c-cf11-4297-bcb0-9bc733c19a28" containerName="payload-processing" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367216 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f04350c-cf11-4297-bcb0-9bc733c19a28" containerName="payload-processing" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367429 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f07eaaa-53cc-4410-bd6e-b8431f435d10" containerName="revoke-keys" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367444 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f04350c-cf11-4297-bcb0-9bc733c19a28" containerName="payload-processing" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367463 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367475 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1173c959-8c68-4379-9ade-818f7d7e8fee" containerName="istio-proxy" Nov 02 20:28:36.368304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.367485 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="173083ea-5a01-4fb6-94ff-7c63d8a6bd10" containerName="payload-pre-processing" Nov 02 20:28:36.412305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.380893 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.412305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.390794 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw"] Nov 02 20:28:36.419690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419654 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-data\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.419858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419747 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-podinfo\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.419858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419779 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-certs\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.419858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419806 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-socket\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.419858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419848 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-envoy\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.420090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419887 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-credential-socket\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.420090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419921 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-token\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.420090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.419977 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba9d419-bf87-4c95-9932-1954286e1a08-istiod-ca-cert\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.420090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.420037 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vc5\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-kube-api-access-29vc5\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.476962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.476932 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-93816f-openshift-default-dockercfg-8zqzw\"" Nov 02 20:28:36.521334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521299 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-credential-socket\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521355 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-token\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521443 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba9d419-bf87-4c95-9932-1954286e1a08-istiod-ca-cert\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521572 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vc5\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-kube-api-access-29vc5\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521712 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-data\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521733 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-credential-socket\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521812 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-podinfo\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521843 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-certs\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.521875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521869 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-socket\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.522041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.521918 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-envoy\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.522041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.522007 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-data\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.522196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.522171 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba9d419-bf87-4c95-9932-1954286e1a08-istiod-ca-cert\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.522264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.522205 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-certs\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.522372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.522266 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-socket\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.524891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.524859 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-envoy\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.525264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.525239 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-podinfo\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.575873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.575831 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-token\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.576032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.575912 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vc5\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-kube-api-access-29vc5\") pod \"e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:36.712043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:36.712007 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:37.158658 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:37.158632 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw"] Nov 02 20:28:37.160127 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:28:37.160101 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba9d419_bf87_4c95_9932_1954286e1a08.slice/crio-54584b9abc34ca24c2b09a98ff6ee339de5f8f122f544dfa04a8340fd94ea508 WatchSource:0}: Error finding container 54584b9abc34ca24c2b09a98ff6ee339de5f8f122f544dfa04a8340fd94ea508: Status 404 returned error can't find the container with id 54584b9abc34ca24c2b09a98ff6ee339de5f8f122f544dfa04a8340fd94ea508 Nov 02 20:28:37.163457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:37.163410 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:37.163535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:37.163503 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:37.163575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:37.163545 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:37.916349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:37.916306 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" event={"ID":"2ba9d419-bf87-4c95-9932-1954286e1a08","Type":"ContainerStarted","Data":"b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0"} Nov 02 20:28:37.916746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:37.916370 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" event={"ID":"2ba9d419-bf87-4c95-9932-1954286e1a08","Type":"ContainerStarted","Data":"54584b9abc34ca24c2b09a98ff6ee339de5f8f122f544dfa04a8340fd94ea508"} Nov 02 20:28:38.029470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:38.029409 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" podStartSLOduration=2.029390603 podStartE2EDuration="2.029390603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:38.02526255 +0000 UTC m=+2416.542035134" watchObservedRunningTime="2026-11-02 20:28:38.029390603 +0000 UTC m=+2416.546163191" Nov 02 20:28:38.713452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:38.713418 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:38.719363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:38.719336 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:38.923652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:38.923620 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:38.924727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:38.924704 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:28:39.589256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.589232 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:28:39.660312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660256 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-data\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660337 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-kube-api-access-qxfqq\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-kube-api-access-qxfqq\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660417 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-token\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-token\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660450 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-certs\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660524 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-credential-socket\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660587 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-data" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:39.660690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660638 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a882e54b-94b4-4453-b990-4a9a64c7d434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e54b-94b4-4453-b990-4a9a64c7d434-istiod-ca-cert\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660753 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-podinfo\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660798 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-socket\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.660866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.660850 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-envoy\") pod \"a882e54b-94b4-4453-b990-4a9a64c7d434\" (UID: \"a882e54b-94b4-4453-b990-4a9a64c7d434\") " Nov 02 20:28:39.661080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.661054 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-credential-socket" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:39.661473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.661446 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.661586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.661477 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.661729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.661707 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-certs" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:39.662081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.662050 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a882e54b-94b4-4453-b990-4a9a64c7d434-istiod-ca-cert" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:28:39.662369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.662345 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-socket" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:39.665036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.664993 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-envoy" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:28:39.665036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.664999 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-podinfo" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:28:39.665159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.665040 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-kube-api-access-qxfqq" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "kube-api-access-qxfqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:39.666212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.666190 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-token" pod "a882e54b-94b4-4453-b990-4a9a64c7d434" (UID: "a882e54b-94b4-4453-b990-4a9a64c7d434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:28:39.763195 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.763164 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxfqq\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-kube-api-access-qxfqq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.763195 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.763193 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.763421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.763203 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.763421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.763213 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e54b-94b4-4453-b990-4a9a64c7d434-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.763421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.763222 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.763421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.763230 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.763421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.763239 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e54b-94b4-4453-b990-4a9a64c7d434-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:28:39.932523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.932435 5800 generic.go:356] "Generic (PLEG): container finished" podID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerID="9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17" exitCode=0 Nov 02 20:28:39.932523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.932488 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" event={"ID":"a882e54b-94b4-4453-b990-4a9a64c7d434","Type":"ContainerDied","Data":"9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17"} Nov 02 20:28:39.933032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.932518 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" Nov 02 20:28:39.933032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.932545 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z" event={"ID":"a882e54b-94b4-4453-b990-4a9a64c7d434","Type":"ContainerDied","Data":"75e9e0523f5937bdd328fbf624ebc9953fe70e9c0cbc7e9a06de72062263f6aa"} Nov 02 20:28:39.933032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.932572 5800 scope.go:122] "RemoveContainer" containerID="9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17" Nov 02 20:28:39.944680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.944647 5800 scope.go:122] "RemoveContainer" containerID="353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c" Nov 02 20:28:39.956887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.956864 5800 scope.go:122] "RemoveContainer" containerID="9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17" Nov 02 20:28:39.957184 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:39.957162 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17\": container with ID starting with 9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17 not found: ID does not exist" containerID="9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17" Nov 02 20:28:39.957273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.957196 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17"} err="failed to get container status \"9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17\": rpc error: code = NotFound desc = could not find container \"9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17\": container with ID starting with 9d6f40483b90429c18c8b03684b8a23509483cf4aae7b4285cec44eec46c9d17 not found: ID does not exist" Nov 02 20:28:39.957273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.957221 5800 scope.go:122] "RemoveContainer" containerID="353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c" Nov 02 20:28:39.957562 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:39.957534 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c\": container with ID starting with 353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c not found: ID does not exist" containerID="353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c" Nov 02 20:28:39.957610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:39.957573 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c"} err="failed to get container status \"353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c\": rpc error: code = NotFound desc = could not find container \"353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c\": container with ID starting with 353fd0b91cc1415be4b76637df11f5c5046d8a6f7dac13642befb8e88a72f81c not found: ID does not exist" Nov 02 20:28:40.086791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:40.086755 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z"] Nov 02 20:28:40.089802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:40.089773 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6f1fd4-openshift-default-67b4bbfd95-kpr2z"] Nov 02 20:28:41.623962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:41.623924 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" path="/var/lib/kubelet/pods/a882e54b-94b4-4453-b990-4a9a64c7d434/volumes" Nov 02 20:28:44.581607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.581550 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz"] Nov 02 20:28:44.590300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.583529 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" Nov 02 20:28:44.590300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.583561 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" Nov 02 20:28:44.590300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.583962 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" Nov 02 20:28:44.590300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.583997 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" Nov 02 20:28:44.593801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.593775 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.654259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.654226 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz"] Nov 02 20:28:44.694207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.694171 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-93816f-dockercfg-cj4gp\"" Nov 02 20:28:44.694423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.694267 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-93816f\"" Nov 02 20:28:44.741342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.739801 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75z57\" (UniqueName: \"kubernetes.io/projected/68a614a3-3987-4614-a716-d188df5b383c-kube-api-access-75z57\") pod \"payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.741342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.739892 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a614a3-3987-4614-a716-d188df5b383c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.766864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.766827 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb"] Nov 02 20:28:44.767990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.767969 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" Nov 02 20:28:44.768133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.768120 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a882e54b-94b4-4453-b990-4a9a64c7d434" containerName="istio-proxy" Nov 02 20:28:44.772424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.772404 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:44.777375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.776467 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb"] Nov 02 20:28:44.849061 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.848968 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:44.849957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.849931 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmtw\" (UniqueName: \"kubernetes.io/projected/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-kube-api-access-7mmtw\") pod \"payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:44.850344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.850320 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75z57\" (UniqueName: \"kubernetes.io/projected/68a614a3-3987-4614-a716-d188df5b383c-kube-api-access-75z57\") pod \"payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.850540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.850519 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a614a3-3987-4614-a716-d188df5b383c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.857174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.857145 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a614a3-3987-4614-a716-d188df5b383c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.904298 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.904255 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75z57\" (UniqueName: \"kubernetes.io/projected/68a614a3-3987-4614-a716-d188df5b383c-kube-api-access-75z57\") pod \"payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.908909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.908881 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:44.956034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.955048 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:44.956333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.956309 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmtw\" (UniqueName: \"kubernetes.io/projected/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-kube-api-access-7mmtw\") pod \"payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:44.957071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:44.955992 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:45.018325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.017079 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmtw\" (UniqueName: \"kubernetes.io/projected/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-kube-api-access-7mmtw\") pod \"payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:45.094428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.094395 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:45.315377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.315344 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz"] Nov 02 20:28:45.316460 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:28:45.316431 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a614a3_3987_4614_a716_d188df5b383c.slice/crio-6c36c8865231cd8c1330f26ec5f7a3e60ce6b23a647f7c26ce25c86b871ce12b WatchSource:0}: Error finding container 6c36c8865231cd8c1330f26ec5f7a3e60ce6b23a647f7c26ce25c86b871ce12b: Status 404 returned error can't find the container with id 6c36c8865231cd8c1330f26ec5f7a3e60ce6b23a647f7c26ce25c86b871ce12b Nov 02 20:28:45.495117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.495081 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb"] Nov 02 20:28:45.496667 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:28:45.496640 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b30d57_44d0_42a7_b69d_2fa8cc1a320e.slice/crio-a01d006c9b049cf936ef691a60a465c729d6bc31cb64c85058c30930a4312a56 WatchSource:0}: Error finding container a01d006c9b049cf936ef691a60a465c729d6bc31cb64c85058c30930a4312a56: Status 404 returned error can't find the container with id a01d006c9b049cf936ef691a60a465c729d6bc31cb64c85058c30930a4312a56 Nov 02 20:28:45.990665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.990620 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" event={"ID":"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e","Type":"ContainerStarted","Data":"c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809"} Nov 02 20:28:45.991133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.990688 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" event={"ID":"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e","Type":"ContainerStarted","Data":"a01d006c9b049cf936ef691a60a465c729d6bc31cb64c85058c30930a4312a56"} Nov 02 20:28:45.991133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.990718 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:45.992330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.992271 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" event={"ID":"68a614a3-3987-4614-a716-d188df5b383c","Type":"ContainerStarted","Data":"65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495"} Nov 02 20:28:45.992451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.992346 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" event={"ID":"68a614a3-3987-4614-a716-d188df5b383c","Type":"ContainerStarted","Data":"6c36c8865231cd8c1330f26ec5f7a3e60ce6b23a647f7c26ce25c86b871ce12b"} Nov 02 20:28:45.992451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:45.992368 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:46.098429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:46.098375 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" podStartSLOduration=2.098349042 podStartE2EDuration="2.098349042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:46.096142723 +0000 UTC m=+2424.612915307" watchObservedRunningTime="2026-11-02 20:28:46.098349042 +0000 UTC m=+2424.615121635" Nov 02 20:28:46.209113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:46.209048 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" podStartSLOduration=2.209027931 podStartE2EDuration="2.209027931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:46.204024743 +0000 UTC m=+2424.720797329" watchObservedRunningTime="2026-11-02 20:28:46.209027931 +0000 UTC m=+2424.725800517" Nov 02 20:28:51.033204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:51.033115 5800 generic.go:356] "Generic (PLEG): container finished" podID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerID="4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2" exitCode=0 Nov 02 20:28:51.033664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:51.033194 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" event={"ID":"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15","Type":"ContainerDied","Data":"4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2"} Nov 02 20:28:51.033728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:51.033689 5800 scope.go:122] "RemoveContainer" containerID="4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2" Nov 02 20:28:51.035528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:51.035497 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:51.035605 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:51.035588 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:51.035643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:51.035633 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:52.062509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:52.062473 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" event={"ID":"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15","Type":"ContainerStarted","Data":"8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c"} Nov 02 20:28:52.696933 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:28:52.696897 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:28:53.054743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:53.054706 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:28:53.054960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:53.054868 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 02 20:28:53.054960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:53.054914 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 02 20:28:54.052978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:54.052940 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 02 20:28:54.053582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:54.052994 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 02 20:28:55.052000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.051966 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 02 20:28:55.052219 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.052020 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 02 20:28:55.657947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.657909 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh"] Nov 02 20:28:55.662324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.662300 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.698961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.698919 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-credential-socket\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.698993 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-data\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.699017 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j79tq\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-kube-api-access-j79tq\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.699064 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c337afa1-a2b2-47fc-accb-4e11c7251b46-istiod-ca-cert\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.699149 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-podinfo\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.699209 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-socket\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.699242 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-certs\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.699342 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-envoy\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.699410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.699393 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-token\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.789452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.789409 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fbe1eb-openshift-default-dockercfg-frdb7\"" Nov 02 20:28:55.802562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.802525 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh"] Nov 02 20:28:55.803736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.803710 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-credential-socket\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.803833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.803783 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-data\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.803833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.803810 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j79tq\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-kube-api-access-j79tq\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.803950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.803862 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c337afa1-a2b2-47fc-accb-4e11c7251b46-istiod-ca-cert\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.803950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.803944 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-podinfo\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.804055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.804042 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-socket\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.804110 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.804070 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-certs\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.804165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.804138 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-envoy\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.804224 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.804195 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-token\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.806412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.805056 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-credential-socket\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.806412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.805338 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-socket\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.806412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.805605 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-certs\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.821745 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.810197 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-data\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.821745 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.810892 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c337afa1-a2b2-47fc-accb-4e11c7251b46-istiod-ca-cert\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.840879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.838543 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-podinfo\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.854664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.850592 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-envoy\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.864634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.864562 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j79tq\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-kube-api-access-j79tq\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.867270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.867214 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-token\") pod \"e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:55.977342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:55.977217 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:56.052721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:56.052677 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 02 20:28:56.052925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:56.052751 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 02 20:28:56.424136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:56.423416 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh"] Nov 02 20:28:56.424512 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:28:56.424485 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc337afa1_a2b2_47fc_accb_4e11c7251b46.slice/crio-fdc15e861ad396b212c80b3a97693c2bb5553a1dc6747b81fc75fca370660536 WatchSource:0}: Error finding container fdc15e861ad396b212c80b3a97693c2bb5553a1dc6747b81fc75fca370660536: Status 404 returned error can't find the container with id fdc15e861ad396b212c80b3a97693c2bb5553a1dc6747b81fc75fca370660536 Nov 02 20:28:56.433304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:56.427962 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:56.433598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:56.433565 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:56.433750 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:56.433732 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:57.001800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.001769 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:28:57.002236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.001839 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:28:57.052823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.052783 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Nov 02 20:28:57.053021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.052857 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Nov 02 20:28:57.097967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.097922 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" event={"ID":"c337afa1-a2b2-47fc-accb-4e11c7251b46","Type":"ContainerStarted","Data":"012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f"} Nov 02 20:28:57.098147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.097990 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" event={"ID":"c337afa1-a2b2-47fc-accb-4e11c7251b46","Type":"ContainerStarted","Data":"fdc15e861ad396b212c80b3a97693c2bb5553a1dc6747b81fc75fca370660536"} Nov 02 20:28:57.360180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.360057 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podStartSLOduration=2.3600415249999998 podStartE2EDuration="2.360041525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:57.359259561 +0000 UTC m=+2435.876032172" watchObservedRunningTime="2026-11-02 20:28:57.360041525 +0000 UTC m=+2435.876814110" Nov 02 20:28:57.977457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.977418 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:57.983478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:57.983453 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:58.056536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:58.056501 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:28:58.057021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:58.056942 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:28:58.057810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:58.057787 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:28:58.109859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:58.109818 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:28:58.110692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:28:58.110669 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:29:02.874820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.874745 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx"] Nov 02 20:29:02.879041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.879023 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:02.928258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.928218 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:29:02.932353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.932325 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx"] Nov 02 20:29:02.944878 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.944846 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9479\" (UniqueName: \"kubernetes.io/projected/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-kube-api-access-p9479\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:02.945054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.944991 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:02.945130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.945082 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:02.974553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.974520 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:29:02.974553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.974526 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:29:02.974760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:02.974576 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:29:03.046258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.046219 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9479\" (UniqueName: \"kubernetes.io/projected/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-kube-api-access-p9479\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:03.046479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.046343 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:03.046479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.046396 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:03.047129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.047104 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:03.048793 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.048764 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:03.099113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.099077 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9479\" (UniqueName: \"kubernetes.io/projected/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-kube-api-access-p9479\") pod \"maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:03.198597 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.198506 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:03.578927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.576981 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx"] Nov 02 20:29:03.578927 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:29:03.578002 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a921b5_4f1c_4f0f_ab8f_aba535d51160.slice/crio-e7fbd0263671c543332c465e5cb432407a868b9fe034ee807092e5ea4e95d651 WatchSource:0}: Error finding container e7fbd0263671c543332c465e5cb432407a868b9fe034ee807092e5ea4e95d651: Status 404 returned error can't find the container with id e7fbd0263671c543332c465e5cb432407a868b9fe034ee807092e5ea4e95d651 Nov 02 20:29:03.952456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.952423 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8"] Nov 02 20:29:03.956832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:03.956813 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.006582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.006553 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fbe1eb\"" Nov 02 20:29:04.006778 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.006595 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fbe1eb-dockercfg-r8zg8\"" Nov 02 20:29:04.009966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.009938 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8"] Nov 02 20:29:04.077302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.077248 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.077472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.077435 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgqz\" (UniqueName: \"kubernetes.io/projected/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-kube-api-access-zxgqz\") pod \"payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.155870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.155829 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" event={"ID":"b4a921b5-4f1c-4f0f-ab8f-aba535d51160","Type":"ContainerStarted","Data":"ee23638cab7c8ce15ddee4e359fbfc722e651f23d6e74b38653db79d3c773de3"} Nov 02 20:29:04.156056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.155891 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" event={"ID":"b4a921b5-4f1c-4f0f-ab8f-aba535d51160","Type":"ContainerStarted","Data":"e7fbd0263671c543332c465e5cb432407a868b9fe034ee807092e5ea4e95d651"} Nov 02 20:29:04.178780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.178737 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.178978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.178892 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgqz\" (UniqueName: \"kubernetes.io/projected/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-kube-api-access-zxgqz\") pod \"payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.179723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.179693 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.232663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.232541 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgqz\" (UniqueName: \"kubernetes.io/projected/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-kube-api-access-zxgqz\") pod \"payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.263463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.263394 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" podStartSLOduration=2.263374984 podStartE2EDuration="2.263374984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:04.261022318 +0000 UTC m=+2442.777794904" watchObservedRunningTime="2026-11-02 20:29:04.263374984 +0000 UTC m=+2442.780147570" Nov 02 20:29:04.271404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.271376 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:04.646323 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:29:04.646271 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aa21ac_a13d_43c9_a9ad_2f4bf0cb8786.slice/crio-727a5a1f27cdee5cf2962a9e7880ae02359d6880a91e423da09caafe5ec7cb7e WatchSource:0}: Error finding container 727a5a1f27cdee5cf2962a9e7880ae02359d6880a91e423da09caafe5ec7cb7e: Status 404 returned error can't find the container with id 727a5a1f27cdee5cf2962a9e7880ae02359d6880a91e423da09caafe5ec7cb7e Nov 02 20:29:04.646494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:04.646326 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8"] Nov 02 20:29:05.166340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:05.166297 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" event={"ID":"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786","Type":"ContainerStarted","Data":"ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a"} Nov 02 20:29:05.166869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:05.166366 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" event={"ID":"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786","Type":"ContainerStarted","Data":"727a5a1f27cdee5cf2962a9e7880ae02359d6880a91e423da09caafe5ec7cb7e"} Nov 02 20:29:05.166869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:05.166397 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:05.167980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:05.167956 5800 generic.go:356] "Generic (PLEG): container finished" podID="b4a921b5-4f1c-4f0f-ab8f-aba535d51160" containerID="ee23638cab7c8ce15ddee4e359fbfc722e651f23d6e74b38653db79d3c773de3" exitCode=0 Nov 02 20:29:05.168130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:05.168030 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" event={"ID":"b4a921b5-4f1c-4f0f-ab8f-aba535d51160","Type":"ContainerDied","Data":"ee23638cab7c8ce15ddee4e359fbfc722e651f23d6e74b38653db79d3c773de3"} Nov 02 20:29:05.274623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:05.274570 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" podStartSLOduration=2.274554363 podStartE2EDuration="2.274554363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:05.272354404 +0000 UTC m=+2443.789126991" watchObservedRunningTime="2026-11-02 20:29:05.274554363 +0000 UTC m=+2443.791326947" Nov 02 20:29:06.361770 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.361741 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:06.509056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.509024 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-kube-api-access-p9479\" (UniqueName: \"kubernetes.io/projected/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-kube-api-access-p9479\") pod \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " Nov 02 20:29:06.509259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.509239 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-openshift-ca\") pod \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " Nov 02 20:29:06.509337 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.509305 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-xks-ca\") pod \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\" (UID: \"b4a921b5-4f1c-4f0f-ab8f-aba535d51160\") " Nov 02 20:29:06.509684 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.509654 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-openshift-ca" pod "b4a921b5-4f1c-4f0f-ab8f-aba535d51160" (UID: "b4a921b5-4f1c-4f0f-ab8f-aba535d51160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:06.509868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.509849 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:06.511408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.511379 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-kube-api-access-p9479" pod "b4a921b5-4f1c-4f0f-ab8f-aba535d51160" (UID: "b4a921b5-4f1c-4f0f-ab8f-aba535d51160"). InnerVolumeSpecName "kube-api-access-p9479". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:06.511798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.511770 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-xks-ca" pod "b4a921b5-4f1c-4f0f-ab8f-aba535d51160" (UID: "b4a921b5-4f1c-4f0f-ab8f-aba535d51160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:29:06.611439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.611403 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9479\" (UniqueName: \"kubernetes.io/projected/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-kube-api-access-p9479\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:06.611439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:06.611433 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a921b5-4f1c-4f0f-ab8f-aba535d51160-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:07.186898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:07.186867 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" Nov 02 20:29:07.187093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:07.186899 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx" event={"ID":"b4a921b5-4f1c-4f0f-ab8f-aba535d51160","Type":"ContainerDied","Data":"e7fbd0263671c543332c465e5cb432407a868b9fe034ee807092e5ea4e95d651"} Nov 02 20:29:07.187093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:07.186950 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7fbd0263671c543332c465e5cb432407a868b9fe034ee807092e5ea4e95d651" Nov 02 20:29:16.182686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:16.182653 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:29:18.778201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:18.778165 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx"] Nov 02 20:29:18.824264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:18.824230 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e87e9a-g5phx"] Nov 02 20:29:19.622792 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:19.622758 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a921b5-4f1c-4f0f-ab8f-aba535d51160" path="/var/lib/kubelet/pods/b4a921b5-4f1c-4f0f-ab8f-aba535d51160/volumes" Nov 02 20:29:22.696447 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:29:22.696411 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:29:34.624971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:34.624940 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z"] Nov 02 20:29:34.723694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:34.723659 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h"] Nov 02 20:29:39.625357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:39.625318 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" podUID="38b428d7-ab0f-480d-bcfe-0779fb106abf" containerName="payload-processing" containerID="cri-o://a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae" gracePeriod=25 Nov 02 20:29:39.724652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:39.724591 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" podUID="944cd387-111f-4550-a4db-bae84d5fea95" containerName="payload-pre-processing" containerID="cri-o://f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9" gracePeriod=25 Nov 02 20:29:39.919324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:39.919047 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:29:39.991889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:39.991862 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:29:40.025517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.025463 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b428d7-ab0f-480d-bcfe-0779fb106abf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b428d7-ab0f-480d-bcfe-0779fb106abf-plugins-config-volume\") pod \"38b428d7-ab0f-480d-bcfe-0779fb106abf\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " Nov 02 20:29:40.025718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.025684 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b428d7-ab0f-480d-bcfe-0779fb106abf-kube-api-access-dd42s\" (UniqueName: \"kubernetes.io/projected/38b428d7-ab0f-480d-bcfe-0779fb106abf-kube-api-access-dd42s\") pod \"38b428d7-ab0f-480d-bcfe-0779fb106abf\" (UID: \"38b428d7-ab0f-480d-bcfe-0779fb106abf\") " Nov 02 20:29:40.025870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.025849 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b428d7-ab0f-480d-bcfe-0779fb106abf-plugins-config-volume" pod "38b428d7-ab0f-480d-bcfe-0779fb106abf" (UID: "38b428d7-ab0f-480d-bcfe-0779fb106abf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:40.026344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.026294 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b428d7-ab0f-480d-bcfe-0779fb106abf-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:40.028533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.028508 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b428d7-ab0f-480d-bcfe-0779fb106abf-kube-api-access-dd42s" pod "38b428d7-ab0f-480d-bcfe-0779fb106abf" (UID: "38b428d7-ab0f-480d-bcfe-0779fb106abf"). InnerVolumeSpecName "kube-api-access-dd42s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:40.126953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.126915 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944cd387-111f-4550-a4db-bae84d5fea95-kube-api-access-xw42p\" (UniqueName: \"kubernetes.io/projected/944cd387-111f-4550-a4db-bae84d5fea95-kube-api-access-xw42p\") pod \"944cd387-111f-4550-a4db-bae84d5fea95\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " Nov 02 20:29:40.127132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.126986 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944cd387-111f-4550-a4db-bae84d5fea95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944cd387-111f-4550-a4db-bae84d5fea95-plugins-config-volume\") pod \"944cd387-111f-4550-a4db-bae84d5fea95\" (UID: \"944cd387-111f-4550-a4db-bae84d5fea95\") " Nov 02 20:29:40.127374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.127357 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd42s\" (UniqueName: \"kubernetes.io/projected/38b428d7-ab0f-480d-bcfe-0779fb106abf-kube-api-access-dd42s\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:40.127450 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.127399 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944cd387-111f-4550-a4db-bae84d5fea95-plugins-config-volume" pod "944cd387-111f-4550-a4db-bae84d5fea95" (UID: "944cd387-111f-4550-a4db-bae84d5fea95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:29:40.129154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.129134 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944cd387-111f-4550-a4db-bae84d5fea95-kube-api-access-xw42p" pod "944cd387-111f-4550-a4db-bae84d5fea95" (UID: "944cd387-111f-4550-a4db-bae84d5fea95"). InnerVolumeSpecName "kube-api-access-xw42p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:29:40.229092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.229006 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw42p\" (UniqueName: \"kubernetes.io/projected/944cd387-111f-4550-a4db-bae84d5fea95-kube-api-access-xw42p\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:40.229092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.229038 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944cd387-111f-4550-a4db-bae84d5fea95-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:29:40.443316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.443263 5800 generic.go:356] "Generic (PLEG): container finished" podID="38b428d7-ab0f-480d-bcfe-0779fb106abf" containerID="a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae" exitCode=0 Nov 02 20:29:40.443503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.443315 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" event={"ID":"38b428d7-ab0f-480d-bcfe-0779fb106abf","Type":"ContainerDied","Data":"a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae"} Nov 02 20:29:40.443503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.443343 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" Nov 02 20:29:40.443503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.443378 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z" event={"ID":"38b428d7-ab0f-480d-bcfe-0779fb106abf","Type":"ContainerDied","Data":"88065f9bfba95c7a22eb21eee7320a0466e8d7e4a40d621088e19659e5e7fef6"} Nov 02 20:29:40.443503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.443407 5800 scope.go:122] "RemoveContainer" containerID="a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae" Nov 02 20:29:40.444854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.444828 5800 generic.go:356] "Generic (PLEG): container finished" podID="944cd387-111f-4550-a4db-bae84d5fea95" containerID="f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9" exitCode=0 Nov 02 20:29:40.444981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.444861 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" event={"ID":"944cd387-111f-4550-a4db-bae84d5fea95","Type":"ContainerDied","Data":"f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9"} Nov 02 20:29:40.444981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.444885 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" Nov 02 20:29:40.444981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.444903 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h" event={"ID":"944cd387-111f-4550-a4db-bae84d5fea95","Type":"ContainerDied","Data":"b3f4b78aab3dfa297afd7baa2ff6247fc43d2d375f835515a968b86a822c1ff7"} Nov 02 20:29:40.455771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.455749 5800 scope.go:122] "RemoveContainer" containerID="a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae" Nov 02 20:29:40.456037 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:29:40.456018 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae\": container with ID starting with a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae not found: ID does not exist" containerID="a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae" Nov 02 20:29:40.456105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.456044 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae"} err="failed to get container status \"a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae\": rpc error: code = NotFound desc = could not find container \"a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae\": container with ID starting with a811fd75b24349fb94160792ef7b0a869e06f24432dbb6b3da70c52c645f6eae not found: ID does not exist" Nov 02 20:29:40.456105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.456061 5800 scope.go:122] "RemoveContainer" containerID="f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9" Nov 02 20:29:40.465066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.465049 5800 scope.go:122] "RemoveContainer" containerID="f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9" Nov 02 20:29:40.465351 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:29:40.465328 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9\": container with ID starting with f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9 not found: ID does not exist" containerID="f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9" Nov 02 20:29:40.465426 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.465361 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9"} err="failed to get container status \"f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9\": rpc error: code = NotFound desc = could not find container \"f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9\": container with ID starting with f4007c4cd36e7fcca5a848dd3194e2e8f291939aebdc3fb6c9f40069d985b1d9 not found: ID does not exist" Nov 02 20:29:40.597642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.597608 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z"] Nov 02 20:29:40.643704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.643668 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b028e6-788cc4848f-hwk2z"] Nov 02 20:29:40.756721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.756678 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h"] Nov 02 20:29:40.760127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:40.760094 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b028e6-7d7c88f467-vlm8h"] Nov 02 20:29:41.625978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:41.625949 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b428d7-ab0f-480d-bcfe-0779fb106abf" path="/var/lib/kubelet/pods/38b428d7-ab0f-480d-bcfe-0779fb106abf/volumes" Nov 02 20:29:41.626485 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:29:41.626469 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944cd387-111f-4550-a4db-bae84d5fea95" path="/var/lib/kubelet/pods/944cd387-111f-4550-a4db-bae84d5fea95/volumes" Nov 02 20:29:52.696943 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:29:52.696897 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:30:00.155905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.155863 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m"] Nov 02 20:30:00.156896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.156836 5800 cpu_manager.go:414] "containerMap: removing container" podUID="944cd387-111f-4550-a4db-bae84d5fea95" containerName="payload-pre-processing" Nov 02 20:30:00.156998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.156899 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944cd387-111f-4550-a4db-bae84d5fea95" containerName="payload-pre-processing" Nov 02 20:30:00.156998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.156950 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b4a921b5-4f1c-4f0f-ab8f-aba535d51160" containerName="revoke-keys" Nov 02 20:30:00.156998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.156959 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a921b5-4f1c-4f0f-ab8f-aba535d51160" containerName="revoke-keys" Nov 02 20:30:00.156998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.156985 5800 cpu_manager.go:414] "containerMap: removing container" podUID="38b428d7-ab0f-480d-bcfe-0779fb106abf" containerName="payload-processing" Nov 02 20:30:00.156998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.156994 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b428d7-ab0f-480d-bcfe-0779fb106abf" containerName="payload-processing" Nov 02 20:30:00.157353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.157201 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a921b5-4f1c-4f0f-ab8f-aba535d51160" containerName="revoke-keys" Nov 02 20:30:00.157353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.157217 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b428d7-ab0f-480d-bcfe-0779fb106abf" containerName="payload-processing" Nov 02 20:30:00.157353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.157234 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="944cd387-111f-4550-a4db-bae84d5fea95" containerName="payload-pre-processing" Nov 02 20:30:00.161478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.161443 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl"] Nov 02 20:30:00.161719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.161695 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.166239 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.166214 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4"] Nov 02 20:30:00.166414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.166380 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.170653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.170626 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.201983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.201950 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr"] Nov 02 20:30:00.206086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.206057 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf"] Nov 02 20:30:00.206247 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.206218 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.210924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.210901 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67"] Nov 02 20:30:00.211065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.211013 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.211352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.211333 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:30:00.211483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.211364 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:30:00.215825 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.215804 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc"] Nov 02 20:30:00.215942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.215923 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.220147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.220127 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.245256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.245227 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:30:00.251261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.251235 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m"] Nov 02 20:30:00.254466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.254441 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl"] Nov 02 20:30:00.257257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.257230 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr"] Nov 02 20:30:00.263905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.263880 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.264024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.263928 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn9zp\" (UniqueName: \"kubernetes.io/projected/bba21cfc-1699-4d9a-90ee-a40b94edbc80-kube-api-access-vn9zp\") pod \"maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.264090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.264069 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba21cfc-1699-4d9a-90ee-a40b94edbc80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.264150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.264132 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03c94f8-a8f9-464e-a350-b5e35da9f839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.264236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.264219 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzl7\" (UniqueName: \"kubernetes.io/projected/d03c94f8-a8f9-464e-a350-b5e35da9f839-kube-api-access-lmzl7\") pod \"maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.264318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.264276 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fs6\" (UniqueName: \"kubernetes.io/projected/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-kube-api-access-76fs6\") pod \"maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.264383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.264371 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxncr\" (UniqueName: \"kubernetes.io/projected/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-kube-api-access-zxncr\") pod \"maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.264465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.264453 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.305290 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.305229 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf"] Nov 02 20:30:00.310554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.310518 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67"] Nov 02 20:30:00.315585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.315553 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc"] Nov 02 20:30:00.357323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.354430 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4"] Nov 02 20:30:00.366129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366086 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e8925-3666-479e-bcb1-770823db1c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.366349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366143 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxncr\" (UniqueName: \"kubernetes.io/projected/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-kube-api-access-zxncr\") pod \"maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.366349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366263 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.366349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366312 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.366537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366368 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn9zp\" (UniqueName: \"kubernetes.io/projected/bba21cfc-1699-4d9a-90ee-a40b94edbc80-kube-api-access-vn9zp\") pod \"maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.366537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366464 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba21cfc-1699-4d9a-90ee-a40b94edbc80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.366537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366520 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f004bee4-4b80-47f3-8468-bd82fcdd12e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.366695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366549 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvn9d\" (UniqueName: \"kubernetes.io/projected/401e8925-3666-479e-bcb1-770823db1c43-kube-api-access-gvn9d\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.366695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366592 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03c94f8-a8f9-464e-a350-b5e35da9f839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.366695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366630 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzl7\" (UniqueName: \"kubernetes.io/projected/d03c94f8-a8f9-464e-a350-b5e35da9f839-kube-api-access-lmzl7\") pod \"maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.366695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366673 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fs6\" (UniqueName: \"kubernetes.io/projected/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-kube-api-access-76fs6\") pod \"maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.366915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366699 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4561427-fd5d-4a8f-bacd-fd1b46c92778-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-5cs67\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.366915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366747 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qllk\" (UniqueName: \"kubernetes.io/projected/f004bee4-4b80-47f3-8468-bd82fcdd12e5-kube-api-access-7qllk\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.366915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.366772 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f426q\" (UniqueName: \"kubernetes.io/projected/f4561427-fd5d-4a8f-bacd-fd1b46c92778-kube-api-access-f426q\") pod \"maas-api-key-cleanup-29894190-5cs67\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.367713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.367667 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba21cfc-1699-4d9a-90ee-a40b94edbc80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.367849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.367730 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.367971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.367953 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.368051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.368027 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03c94f8-a8f9-464e-a350-b5e35da9f839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.420209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.420130 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn9zp\" (UniqueName: \"kubernetes.io/projected/bba21cfc-1699-4d9a-90ee-a40b94edbc80-kube-api-access-vn9zp\") pod \"maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.420209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.420158 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzl7\" (UniqueName: \"kubernetes.io/projected/d03c94f8-a8f9-464e-a350-b5e35da9f839-kube-api-access-lmzl7\") pod \"maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.420428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.420378 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxncr\" (UniqueName: \"kubernetes.io/projected/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-kube-api-access-zxncr\") pod \"maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.420474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.420442 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fs6\" (UniqueName: \"kubernetes.io/projected/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-kube-api-access-76fs6\") pod \"maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.468500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.468445 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f004bee4-4b80-47f3-8468-bd82fcdd12e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.468500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.468510 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn9d\" (UniqueName: \"kubernetes.io/projected/401e8925-3666-479e-bcb1-770823db1c43-kube-api-access-gvn9d\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.468775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.468574 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4561427-fd5d-4a8f-bacd-fd1b46c92778-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-5cs67\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.468775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.468617 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qllk\" (UniqueName: \"kubernetes.io/projected/f004bee4-4b80-47f3-8468-bd82fcdd12e5-kube-api-access-7qllk\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.468775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.468644 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f426q\" (UniqueName: \"kubernetes.io/projected/f4561427-fd5d-4a8f-bacd-fd1b46c92778-kube-api-access-f426q\") pod \"maas-api-key-cleanup-29894190-5cs67\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.468775 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.468700 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e8925-3666-479e-bcb1-770823db1c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.469342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.469313 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4561427-fd5d-4a8f-bacd-fd1b46c92778-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894190-5cs67\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.469439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.469323 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f004bee4-4b80-47f3-8468-bd82fcdd12e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.469486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.469463 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e8925-3666-479e-bcb1-770823db1c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.479796 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.479773 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:00.490995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.490969 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:00.497603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.497580 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:00.522379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.522338 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f426q\" (UniqueName: \"kubernetes.io/projected/f4561427-fd5d-4a8f-bacd-fd1b46c92778-kube-api-access-f426q\") pod \"maas-api-key-cleanup-29894190-5cs67\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.523339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.523218 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvn9d\" (UniqueName: \"kubernetes.io/projected/401e8925-3666-479e-bcb1-770823db1c43-kube-api-access-gvn9d\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.525059 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.524882 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:00.526885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.526862 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qllk\" (UniqueName: \"kubernetes.io/projected/f004bee4-4b80-47f3-8468-bd82fcdd12e5-kube-api-access-7qllk\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.534699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.534138 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:00.548462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.544525 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:00.551502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.551243 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:00.876436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.876409 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m"] Nov 02 20:30:00.876742 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:00.876711 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbfb1f2_5486_4b4c_8390_f2ff2a0c7c6c.slice/crio-95cd51c4bf0a493226164f3d7d29cd346d5363f71673a7d992d6bd551006bf92 WatchSource:0}: Error finding container 95cd51c4bf0a493226164f3d7d29cd346d5363f71673a7d992d6bd551006bf92: Status 404 returned error can't find the container with id 95cd51c4bf0a493226164f3d7d29cd346d5363f71673a7d992d6bd551006bf92 Nov 02 20:30:00.920666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.920638 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl"] Nov 02 20:30:00.921615 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:00.921590 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae7e43b_a128_45e0_b2e5_f32b4350d81c.slice/crio-497b405bbc5dbcbf3210ccbc9feeeed85dc04b16e702bfb93b8856852ec0e0a4 WatchSource:0}: Error finding container 497b405bbc5dbcbf3210ccbc9feeeed85dc04b16e702bfb93b8856852ec0e0a4: Status 404 returned error can't find the container with id 497b405bbc5dbcbf3210ccbc9feeeed85dc04b16e702bfb93b8856852ec0e0a4 Nov 02 20:30:00.974512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.974481 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4"] Nov 02 20:30:00.975031 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:00.975007 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba21cfc_1699_4d9a_90ee_a40b94edbc80.slice/crio-1dfbf0b17898c40c3cf8a43f7258415756a675111aeee2ab3e5934ba8d9433cf WatchSource:0}: Error finding container 1dfbf0b17898c40c3cf8a43f7258415756a675111aeee2ab3e5934ba8d9433cf: Status 404 returned error can't find the container with id 1dfbf0b17898c40c3cf8a43f7258415756a675111aeee2ab3e5934ba8d9433cf Nov 02 20:30:00.987594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:00.987569 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr"] Nov 02 20:30:00.988058 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:00.988034 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03c94f8_a8f9_464e_a350_b5e35da9f839.slice/crio-63273a0f7c4d44ab1e7b420065a8a367e51b61c088109224ef4a7155471e5c33 WatchSource:0}: Error finding container 63273a0f7c4d44ab1e7b420065a8a367e51b61c088109224ef4a7155471e5c33: Status 404 returned error can't find the container with id 63273a0f7c4d44ab1e7b420065a8a367e51b61c088109224ef4a7155471e5c33 Nov 02 20:30:01.043807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.043765 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf"] Nov 02 20:30:01.055557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.055530 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc"] Nov 02 20:30:01.059642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.059618 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67"] Nov 02 20:30:01.129745 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:01.129706 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf004bee4_4b80_47f3_8468_bd82fcdd12e5.slice/crio-34100d9913934851f87809f872977fe67a30005fd85b62e9b5c66702088cbffa WatchSource:0}: Error finding container 34100d9913934851f87809f872977fe67a30005fd85b62e9b5c66702088cbffa: Status 404 returned error can't find the container with id 34100d9913934851f87809f872977fe67a30005fd85b62e9b5c66702088cbffa Nov 02 20:30:01.139482 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:01.138978 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4561427_fd5d_4a8f_bacd_fd1b46c92778.slice/crio-f25185d3489450a001d26e39f94d04c252f8b29135f48623cc91121560fdd21f WatchSource:0}: Error finding container f25185d3489450a001d26e39f94d04c252f8b29135f48623cc91121560fdd21f: Status 404 returned error can't find the container with id f25185d3489450a001d26e39f94d04c252f8b29135f48623cc91121560fdd21f Nov 02 20:30:01.658064 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.658029 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" event={"ID":"f004bee4-4b80-47f3-8468-bd82fcdd12e5","Type":"ContainerStarted","Data":"fb5eb43084a7e9c2809f603320cde2f86d2ec9908ee481dcd191599c86cd4187"} Nov 02 20:30:01.658562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.658083 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" event={"ID":"f004bee4-4b80-47f3-8468-bd82fcdd12e5","Type":"ContainerStarted","Data":"34100d9913934851f87809f872977fe67a30005fd85b62e9b5c66702088cbffa"} Nov 02 20:30:01.660185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.660160 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" event={"ID":"bba21cfc-1699-4d9a-90ee-a40b94edbc80","Type":"ContainerStarted","Data":"4d9a6bfd4221abd1d4c58b5c03a340ae7064b9754170bc4c53e60e36c0b7268f"} Nov 02 20:30:01.660342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.660212 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" event={"ID":"bba21cfc-1699-4d9a-90ee-a40b94edbc80","Type":"ContainerStarted","Data":"1dfbf0b17898c40c3cf8a43f7258415756a675111aeee2ab3e5934ba8d9433cf"} Nov 02 20:30:01.661893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.661868 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" event={"ID":"d03c94f8-a8f9-464e-a350-b5e35da9f839","Type":"ContainerStarted","Data":"20fbffd62e457359ebfd537fa7d4452dc350aa8a6789345757a26f340cd2136f"} Nov 02 20:30:01.662000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.661914 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" event={"ID":"d03c94f8-a8f9-464e-a350-b5e35da9f839","Type":"ContainerStarted","Data":"63273a0f7c4d44ab1e7b420065a8a367e51b61c088109224ef4a7155471e5c33"} Nov 02 20:30:01.663697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.663676 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" event={"ID":"3ae7e43b-a128-45e0-b2e5-f32b4350d81c","Type":"ContainerStarted","Data":"355fab42db44b4769f401ce672748ffcfff2339ecc30ea36e88b035fd62aee91"} Nov 02 20:30:01.663800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.663716 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" event={"ID":"3ae7e43b-a128-45e0-b2e5-f32b4350d81c","Type":"ContainerStarted","Data":"497b405bbc5dbcbf3210ccbc9feeeed85dc04b16e702bfb93b8856852ec0e0a4"} Nov 02 20:30:01.665435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.665407 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" event={"ID":"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c","Type":"ContainerStarted","Data":"626ceb9cd93d9047a796109fbfc0cc51ac0e23a5abdea10105a133e7d519d4e7"} Nov 02 20:30:01.665531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.665467 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" event={"ID":"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c","Type":"ContainerStarted","Data":"95cd51c4bf0a493226164f3d7d29cd346d5363f71673a7d992d6bd551006bf92"} Nov 02 20:30:01.667071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.667050 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" event={"ID":"f4561427-fd5d-4a8f-bacd-fd1b46c92778","Type":"ContainerStarted","Data":"9641e18425f8f6e4808b2a29779b09110db256d4a60064c3153e8ce5ab37b9ad"} Nov 02 20:30:01.667185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.667091 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" event={"ID":"f4561427-fd5d-4a8f-bacd-fd1b46c92778","Type":"ContainerStarted","Data":"f25185d3489450a001d26e39f94d04c252f8b29135f48623cc91121560fdd21f"} Nov 02 20:30:01.668331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.668312 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" event={"ID":"401e8925-3666-479e-bcb1-770823db1c43","Type":"ContainerStarted","Data":"0899d3719844de17977cab5304e1f0b1844641d6c4d6c51ee8f2226f1885586d"} Nov 02 20:30:01.768651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.768586 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" podStartSLOduration=1.768567801 podStartE2EDuration="1.768567801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:01.762837835 +0000 UTC m=+2500.279610432" watchObservedRunningTime="2026-11-02 20:30:01.768567801 +0000 UTC m=+2500.285340380" Nov 02 20:30:01.871670 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.871595 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" podStartSLOduration=1.87157633 podStartE2EDuration="1.87157633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:01.866859127 +0000 UTC m=+2500.383631713" watchObservedRunningTime="2026-11-02 20:30:01.87157633 +0000 UTC m=+2500.388348915" Nov 02 20:30:01.973140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:01.973088 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" podStartSLOduration=1.973072087 podStartE2EDuration="1.973072087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:01.971369608 +0000 UTC m=+2500.488142193" watchObservedRunningTime="2026-11-02 20:30:01.973072087 +0000 UTC m=+2500.489844733" Nov 02 20:30:02.079451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.079390 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" podStartSLOduration=2.079374922 podStartE2EDuration="2.079374922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.077109079 +0000 UTC m=+2500.593881665" watchObservedRunningTime="2026-11-02 20:30:02.079374922 +0000 UTC m=+2500.596147518" Nov 02 20:30:02.182638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.182533 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" podStartSLOduration=2.182519915 podStartE2EDuration="2.182519915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.181638478 +0000 UTC m=+2500.698411063" watchObservedRunningTime="2026-11-02 20:30:02.182519915 +0000 UTC m=+2500.699292500" Nov 02 20:30:02.289572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.289525 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" podStartSLOduration=2.2895108889999998 podStartE2EDuration="2.289510889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:02.287553496 +0000 UTC m=+2500.804326082" watchObservedRunningTime="2026-11-02 20:30:02.289510889 +0000 UTC m=+2500.806283474" Nov 02 20:30:02.677789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.677753 5800 generic.go:356] "Generic (PLEG): container finished" podID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerID="9641e18425f8f6e4808b2a29779b09110db256d4a60064c3153e8ce5ab37b9ad" exitCode=60 Nov 02 20:30:02.678235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.677831 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" event={"ID":"f4561427-fd5d-4a8f-bacd-fd1b46c92778","Type":"ContainerDied","Data":"9641e18425f8f6e4808b2a29779b09110db256d4a60064c3153e8ce5ab37b9ad"} Nov 02 20:30:02.678235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.678149 5800 scope.go:122] "RemoveContainer" containerID="9641e18425f8f6e4808b2a29779b09110db256d4a60064c3153e8ce5ab37b9ad" Nov 02 20:30:02.679576 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.679553 5800 generic.go:356] "Generic (PLEG): container finished" podID="401e8925-3666-479e-bcb1-770823db1c43" containerID="c17fb55d9c505373067dc12a3adcf38111249b90392fb2dc9b443f7ad521e088" exitCode=60 Nov 02 20:30:02.679663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.679628 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" event={"ID":"401e8925-3666-479e-bcb1-770823db1c43","Type":"ContainerDied","Data":"c17fb55d9c505373067dc12a3adcf38111249b90392fb2dc9b443f7ad521e088"} Nov 02 20:30:02.679849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.679834 5800 scope.go:122] "RemoveContainer" containerID="c17fb55d9c505373067dc12a3adcf38111249b90392fb2dc9b443f7ad521e088" Nov 02 20:30:02.681530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.681495 5800 generic.go:356] "Generic (PLEG): container finished" podID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerID="fb5eb43084a7e9c2809f603320cde2f86d2ec9908ee481dcd191599c86cd4187" exitCode=60 Nov 02 20:30:02.681600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.681577 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" event={"ID":"f004bee4-4b80-47f3-8468-bd82fcdd12e5","Type":"ContainerDied","Data":"fb5eb43084a7e9c2809f603320cde2f86d2ec9908ee481dcd191599c86cd4187"} Nov 02 20:30:02.681771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.681753 5800 scope.go:122] "RemoveContainer" containerID="fb5eb43084a7e9c2809f603320cde2f86d2ec9908ee481dcd191599c86cd4187" Nov 02 20:30:02.683174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.683104 5800 generic.go:356] "Generic (PLEG): container finished" podID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerID="4d9a6bfd4221abd1d4c58b5c03a340ae7064b9754170bc4c53e60e36c0b7268f" exitCode=60 Nov 02 20:30:02.683174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.683143 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" event={"ID":"bba21cfc-1699-4d9a-90ee-a40b94edbc80","Type":"ContainerDied","Data":"4d9a6bfd4221abd1d4c58b5c03a340ae7064b9754170bc4c53e60e36c0b7268f"} Nov 02 20:30:02.683447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.683429 5800 scope.go:122] "RemoveContainer" containerID="4d9a6bfd4221abd1d4c58b5c03a340ae7064b9754170bc4c53e60e36c0b7268f" Nov 02 20:30:02.684776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.684758 5800 generic.go:356] "Generic (PLEG): container finished" podID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerID="20fbffd62e457359ebfd537fa7d4452dc350aa8a6789345757a26f340cd2136f" exitCode=60 Nov 02 20:30:02.684879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.684840 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" event={"ID":"d03c94f8-a8f9-464e-a350-b5e35da9f839","Type":"ContainerDied","Data":"20fbffd62e457359ebfd537fa7d4452dc350aa8a6789345757a26f340cd2136f"} Nov 02 20:30:02.684998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.684984 5800 scope.go:122] "RemoveContainer" containerID="20fbffd62e457359ebfd537fa7d4452dc350aa8a6789345757a26f340cd2136f" Nov 02 20:30:02.686413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.686392 5800 generic.go:356] "Generic (PLEG): container finished" podID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerID="355fab42db44b4769f401ce672748ffcfff2339ecc30ea36e88b035fd62aee91" exitCode=60 Nov 02 20:30:02.686508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.686474 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" event={"ID":"3ae7e43b-a128-45e0-b2e5-f32b4350d81c","Type":"ContainerDied","Data":"355fab42db44b4769f401ce672748ffcfff2339ecc30ea36e88b035fd62aee91"} Nov 02 20:30:02.686640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.686624 5800 scope.go:122] "RemoveContainer" containerID="355fab42db44b4769f401ce672748ffcfff2339ecc30ea36e88b035fd62aee91" Nov 02 20:30:02.688152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.688129 5800 generic.go:356] "Generic (PLEG): container finished" podID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerID="626ceb9cd93d9047a796109fbfc0cc51ac0e23a5abdea10105a133e7d519d4e7" exitCode=60 Nov 02 20:30:02.688234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.688185 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" event={"ID":"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c","Type":"ContainerDied","Data":"626ceb9cd93d9047a796109fbfc0cc51ac0e23a5abdea10105a133e7d519d4e7"} Nov 02 20:30:02.688400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:02.688386 5800 scope.go:122] "RemoveContainer" containerID="626ceb9cd93d9047a796109fbfc0cc51ac0e23a5abdea10105a133e7d519d4e7" Nov 02 20:30:03.707799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:03.707162 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" event={"ID":"f4561427-fd5d-4a8f-bacd-fd1b46c92778","Type":"ContainerStarted","Data":"8e8ba78db0b9088f4987d276f4c879c69bdb1fec21617b427e6dd8eea64baf8c"} Nov 02 20:30:03.711564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:03.711534 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" event={"ID":"401e8925-3666-479e-bcb1-770823db1c43","Type":"ContainerStarted","Data":"94848f62a268262946d228771545353c417698cfcbd7bc592aa732cedbab4f82"} Nov 02 20:30:03.713962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:03.713935 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" event={"ID":"f004bee4-4b80-47f3-8468-bd82fcdd12e5","Type":"ContainerStarted","Data":"aa4febc464401943dcdc0ae1322e79432da0111249367c0c6eb6b112a247f04e"} Nov 02 20:30:03.716089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:03.716064 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" event={"ID":"bba21cfc-1699-4d9a-90ee-a40b94edbc80","Type":"ContainerStarted","Data":"541a3fd4dd2bc71d3f15be36d4e116ce36a5980cc1f7c1e41135b227d3efb38a"} Nov 02 20:30:03.718024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:03.718001 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" event={"ID":"d03c94f8-a8f9-464e-a350-b5e35da9f839","Type":"ContainerStarted","Data":"dea7b71d537197c4264584058d2d82ed16b8a093ee402e6d230ed0490cc5c679"} Nov 02 20:30:03.719873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:03.719852 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" event={"ID":"3ae7e43b-a128-45e0-b2e5-f32b4350d81c","Type":"ContainerStarted","Data":"d128fd3de319afd3ad3a19b9fec08a5d61e8b3091f41f7030a2e30e639d1a059"} Nov 02 20:30:03.721636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:03.721612 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" event={"ID":"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c","Type":"ContainerStarted","Data":"051a59f3bca26ae9edead31cc81c7aa5f2ca041aed1b6c3a2170083767ef8cc1"} Nov 02 20:30:04.342014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.341954 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" podStartSLOduration=4.341940201 podStartE2EDuration="4.341940201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:04.340760456 +0000 UTC m=+2502.857533041" watchObservedRunningTime="2026-11-02 20:30:04.341940201 +0000 UTC m=+2502.858712786" Nov 02 20:30:04.732074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.732041 5800 generic.go:356] "Generic (PLEG): container finished" podID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerID="051a59f3bca26ae9edead31cc81c7aa5f2ca041aed1b6c3a2170083767ef8cc1" exitCode=60 Nov 02 20:30:04.732675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.732129 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" event={"ID":"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c","Type":"ContainerDied","Data":"051a59f3bca26ae9edead31cc81c7aa5f2ca041aed1b6c3a2170083767ef8cc1"} Nov 02 20:30:04.732675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.732198 5800 scope.go:122] "RemoveContainer" containerID="626ceb9cd93d9047a796109fbfc0cc51ac0e23a5abdea10105a133e7d519d4e7" Nov 02 20:30:04.732675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.732586 5800 scope.go:122] "RemoveContainer" containerID="051a59f3bca26ae9edead31cc81c7aa5f2ca041aed1b6c3a2170083767ef8cc1" Nov 02 20:30:04.732966 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:04.732932 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m_odh-ai-gateway-infra(8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" Nov 02 20:30:04.734715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.734690 5800 generic.go:356] "Generic (PLEG): container finished" podID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerID="8e8ba78db0b9088f4987d276f4c879c69bdb1fec21617b427e6dd8eea64baf8c" exitCode=60 Nov 02 20:30:04.734856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.734773 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" event={"ID":"f4561427-fd5d-4a8f-bacd-fd1b46c92778","Type":"ContainerDied","Data":"8e8ba78db0b9088f4987d276f4c879c69bdb1fec21617b427e6dd8eea64baf8c"} Nov 02 20:30:04.735009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.734993 5800 scope.go:122] "RemoveContainer" containerID="8e8ba78db0b9088f4987d276f4c879c69bdb1fec21617b427e6dd8eea64baf8c" Nov 02 20:30:04.735397 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:04.735230 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-5cs67_odh-ai-gateway-infra(f4561427-fd5d-4a8f-bacd-fd1b46c92778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" Nov 02 20:30:04.737079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.737057 5800 generic.go:356] "Generic (PLEG): container finished" podID="401e8925-3666-479e-bcb1-770823db1c43" containerID="94848f62a268262946d228771545353c417698cfcbd7bc592aa732cedbab4f82" exitCode=60 Nov 02 20:30:04.737375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.737254 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" event={"ID":"401e8925-3666-479e-bcb1-770823db1c43","Type":"ContainerDied","Data":"94848f62a268262946d228771545353c417698cfcbd7bc592aa732cedbab4f82"} Nov 02 20:30:04.737520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.737504 5800 scope.go:122] "RemoveContainer" containerID="94848f62a268262946d228771545353c417698cfcbd7bc592aa732cedbab4f82" Nov 02 20:30:04.738197 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:04.737771 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc_odh-ai-gateway-infra(401e8925-3666-479e-bcb1-770823db1c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" podUID="401e8925-3666-479e-bcb1-770823db1c43" Nov 02 20:30:04.739815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.739778 5800 generic.go:356] "Generic (PLEG): container finished" podID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerID="aa4febc464401943dcdc0ae1322e79432da0111249367c0c6eb6b112a247f04e" exitCode=60 Nov 02 20:30:04.739922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.739872 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" event={"ID":"f004bee4-4b80-47f3-8468-bd82fcdd12e5","Type":"ContainerDied","Data":"aa4febc464401943dcdc0ae1322e79432da0111249367c0c6eb6b112a247f04e"} Nov 02 20:30:04.740120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.740101 5800 scope.go:122] "RemoveContainer" containerID="aa4febc464401943dcdc0ae1322e79432da0111249367c0c6eb6b112a247f04e" Nov 02 20:30:04.740434 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:04.740409 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf_odh-ai-gateway-infra(f004bee4-4b80-47f3-8468-bd82fcdd12e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.742445 5800 generic.go:356] "Generic (PLEG): container finished" podID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerID="541a3fd4dd2bc71d3f15be36d4e116ce36a5980cc1f7c1e41135b227d3efb38a" exitCode=60 Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.742719 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" event={"ID":"bba21cfc-1699-4d9a-90ee-a40b94edbc80","Type":"ContainerDied","Data":"541a3fd4dd2bc71d3f15be36d4e116ce36a5980cc1f7c1e41135b227d3efb38a"} Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.742882 5800 scope.go:122] "RemoveContainer" containerID="541a3fd4dd2bc71d3f15be36d4e116ce36a5980cc1f7c1e41135b227d3efb38a" Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:04.743127 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4_odh-ai-gateway-infra(bba21cfc-1699-4d9a-90ee-a40b94edbc80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.744876 5800 generic.go:356] "Generic (PLEG): container finished" podID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerID="dea7b71d537197c4264584058d2d82ed16b8a093ee402e6d230ed0490cc5c679" exitCode=60 Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.744939 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" event={"ID":"d03c94f8-a8f9-464e-a350-b5e35da9f839","Type":"ContainerDied","Data":"dea7b71d537197c4264584058d2d82ed16b8a093ee402e6d230ed0490cc5c679"} Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.745229 5800 scope.go:122] "RemoveContainer" containerID="dea7b71d537197c4264584058d2d82ed16b8a093ee402e6d230ed0490cc5c679" Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:04.745534 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr_odh-ai-gateway-infra(d03c94f8-a8f9-464e-a350-b5e35da9f839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.747416 5800 generic.go:356] "Generic (PLEG): container finished" podID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerID="d128fd3de319afd3ad3a19b9fec08a5d61e8b3091f41f7030a2e30e639d1a059" exitCode=60 Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.747456 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" event={"ID":"3ae7e43b-a128-45e0-b2e5-f32b4350d81c","Type":"ContainerDied","Data":"d128fd3de319afd3ad3a19b9fec08a5d61e8b3091f41f7030a2e30e639d1a059"} Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.747688 5800 scope.go:122] "RemoveContainer" containerID="d128fd3de319afd3ad3a19b9fec08a5d61e8b3091f41f7030a2e30e639d1a059" Nov 02 20:30:04.748321 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:04.747984 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl_odh-ai-gateway-infra(3ae7e43b-a128-45e0-b2e5-f32b4350d81c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" Nov 02 20:30:04.748949 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.748691 5800 scope.go:122] "RemoveContainer" containerID="9641e18425f8f6e4808b2a29779b09110db256d4a60064c3153e8ce5ab37b9ad" Nov 02 20:30:04.765585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.765553 5800 scope.go:122] "RemoveContainer" containerID="c17fb55d9c505373067dc12a3adcf38111249b90392fb2dc9b443f7ad521e088" Nov 02 20:30:04.779349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.779270 5800 scope.go:122] "RemoveContainer" containerID="fb5eb43084a7e9c2809f603320cde2f86d2ec9908ee481dcd191599c86cd4187" Nov 02 20:30:04.797525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.797500 5800 scope.go:122] "RemoveContainer" containerID="4d9a6bfd4221abd1d4c58b5c03a340ae7064b9754170bc4c53e60e36c0b7268f" Nov 02 20:30:04.812653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.812631 5800 scope.go:122] "RemoveContainer" containerID="20fbffd62e457359ebfd537fa7d4452dc350aa8a6789345757a26f340cd2136f" Nov 02 20:30:04.828025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:04.828000 5800 scope.go:122] "RemoveContainer" containerID="355fab42db44b4769f401ce672748ffcfff2339ecc30ea36e88b035fd62aee91" Nov 02 20:30:05.764588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:05.764555 5800 scope.go:122] "RemoveContainer" containerID="aa4febc464401943dcdc0ae1322e79432da0111249367c0c6eb6b112a247f04e" Nov 02 20:30:05.765064 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:05.764899 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf_odh-ai-gateway-infra(f004bee4-4b80-47f3-8468-bd82fcdd12e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" Nov 02 20:30:05.766669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:05.766645 5800 scope.go:122] "RemoveContainer" containerID="541a3fd4dd2bc71d3f15be36d4e116ce36a5980cc1f7c1e41135b227d3efb38a" Nov 02 20:30:05.766954 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:05.766928 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4_odh-ai-gateway-infra(bba21cfc-1699-4d9a-90ee-a40b94edbc80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" Nov 02 20:30:05.768539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:05.768516 5800 scope.go:122] "RemoveContainer" containerID="dea7b71d537197c4264584058d2d82ed16b8a093ee402e6d230ed0490cc5c679" Nov 02 20:30:05.768817 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:05.768794 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr_odh-ai-gateway-infra(d03c94f8-a8f9-464e-a350-b5e35da9f839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" Nov 02 20:30:05.770250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:05.770227 5800 scope.go:122] "RemoveContainer" containerID="d128fd3de319afd3ad3a19b9fec08a5d61e8b3091f41f7030a2e30e639d1a059" Nov 02 20:30:05.770508 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:05.770485 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl_odh-ai-gateway-infra(3ae7e43b-a128-45e0-b2e5-f32b4350d81c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" Nov 02 20:30:05.771880 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:05.771861 5800 scope.go:122] "RemoveContainer" containerID="051a59f3bca26ae9edead31cc81c7aa5f2ca041aed1b6c3a2170083767ef8cc1" Nov 02 20:30:05.772098 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:05.772076 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m_odh-ai-gateway-infra(8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" Nov 02 20:30:05.773580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:05.773563 5800 scope.go:122] "RemoveContainer" containerID="8e8ba78db0b9088f4987d276f4c879c69bdb1fec21617b427e6dd8eea64baf8c" Nov 02 20:30:05.773781 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:05.773757 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-5cs67_odh-ai-gateway-infra(f4561427-fd5d-4a8f-bacd-fd1b46c92778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" Nov 02 20:30:05.775210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:05.775192 5800 scope.go:122] "RemoveContainer" containerID="94848f62a268262946d228771545353c417698cfcbd7bc592aa732cedbab4f82" Nov 02 20:30:05.775465 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:05.775444 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc_odh-ai-gateway-infra(401e8925-3666-479e-bcb1-770823db1c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" podUID="401e8925-3666-479e-bcb1-770823db1c43" Nov 02 20:30:06.236315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:06.236197 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5"] Nov 02 20:30:06.337804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:06.337763 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c"] Nov 02 20:30:11.239341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.238345 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" podUID="e3c504e0-833e-41f5-899a-1fb20e1d2a50" containerName="payload-processing" containerID="cri-o://fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380" gracePeriod=25 Nov 02 20:30:11.355305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.347399 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" podUID="25aaa18e-437d-4149-a284-e06b174ab477" containerName="payload-pre-processing" containerID="cri-o://b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024" gracePeriod=25 Nov 02 20:30:11.601495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.601469 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:30:11.668944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.668909 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c504e0-833e-41f5-899a-1fb20e1d2a50-kube-api-access-gt67n\" (UniqueName: \"kubernetes.io/projected/e3c504e0-833e-41f5-899a-1fb20e1d2a50-kube-api-access-gt67n\") pod \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " Nov 02 20:30:11.669104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.669001 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c504e0-833e-41f5-899a-1fb20e1d2a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c504e0-833e-41f5-899a-1fb20e1d2a50-plugins-config-volume\") pod \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\" (UID: \"e3c504e0-833e-41f5-899a-1fb20e1d2a50\") " Nov 02 20:30:11.670767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.670734 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c504e0-833e-41f5-899a-1fb20e1d2a50-plugins-config-volume" pod "e3c504e0-833e-41f5-899a-1fb20e1d2a50" (UID: "e3c504e0-833e-41f5-899a-1fb20e1d2a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:11.679644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.679616 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c504e0-833e-41f5-899a-1fb20e1d2a50-kube-api-access-gt67n" pod "e3c504e0-833e-41f5-899a-1fb20e1d2a50" (UID: "e3c504e0-833e-41f5-899a-1fb20e1d2a50"). InnerVolumeSpecName "kube-api-access-gt67n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:11.720102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.720075 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:30:11.770820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.770788 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25aaa18e-437d-4149-a284-e06b174ab477-kube-api-access-bmzrk\" (UniqueName: \"kubernetes.io/projected/25aaa18e-437d-4149-a284-e06b174ab477-kube-api-access-bmzrk\") pod \"25aaa18e-437d-4149-a284-e06b174ab477\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " Nov 02 20:30:11.771001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.770936 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25aaa18e-437d-4149-a284-e06b174ab477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aaa18e-437d-4149-a284-e06b174ab477-plugins-config-volume\") pod \"25aaa18e-437d-4149-a284-e06b174ab477\" (UID: \"25aaa18e-437d-4149-a284-e06b174ab477\") " Nov 02 20:30:11.771352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.771324 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25aaa18e-437d-4149-a284-e06b174ab477-plugins-config-volume" pod "25aaa18e-437d-4149-a284-e06b174ab477" (UID: "25aaa18e-437d-4149-a284-e06b174ab477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:11.771605 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.771587 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25aaa18e-437d-4149-a284-e06b174ab477-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:11.771674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.771612 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt67n\" (UniqueName: \"kubernetes.io/projected/e3c504e0-833e-41f5-899a-1fb20e1d2a50-kube-api-access-gt67n\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:11.771674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.771626 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c504e0-833e-41f5-899a-1fb20e1d2a50-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:11.773151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.773128 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25aaa18e-437d-4149-a284-e06b174ab477-kube-api-access-bmzrk" pod "25aaa18e-437d-4149-a284-e06b174ab477" (UID: "25aaa18e-437d-4149-a284-e06b174ab477"). InnerVolumeSpecName "kube-api-access-bmzrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:11.872776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.872733 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzrk\" (UniqueName: \"kubernetes.io/projected/25aaa18e-437d-4149-a284-e06b174ab477-kube-api-access-bmzrk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:11.880390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.880352 5800 generic.go:356] "Generic (PLEG): container finished" podID="25aaa18e-437d-4149-a284-e06b174ab477" containerID="b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024" exitCode=0 Nov 02 20:30:11.880650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.880407 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" Nov 02 20:30:11.880650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.880426 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" event={"ID":"25aaa18e-437d-4149-a284-e06b174ab477","Type":"ContainerDied","Data":"b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024"} Nov 02 20:30:11.880650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.880483 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c" event={"ID":"25aaa18e-437d-4149-a284-e06b174ab477","Type":"ContainerDied","Data":"dee35a7ea3feb1f39b702889759a782722af991814e4b0445874fcb42a1f1568"} Nov 02 20:30:11.880650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.880513 5800 scope.go:122] "RemoveContainer" containerID="b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024" Nov 02 20:30:11.882580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.882555 5800 generic.go:356] "Generic (PLEG): container finished" podID="e3c504e0-833e-41f5-899a-1fb20e1d2a50" containerID="fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380" exitCode=0 Nov 02 20:30:11.882697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.882628 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" event={"ID":"e3c504e0-833e-41f5-899a-1fb20e1d2a50","Type":"ContainerDied","Data":"fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380"} Nov 02 20:30:11.882697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.882683 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" event={"ID":"e3c504e0-833e-41f5-899a-1fb20e1d2a50","Type":"ContainerDied","Data":"5f1ba6f97b9b34dc489420e0a856b88bd0c112cb63e73755e2b49aed5aa65300"} Nov 02 20:30:11.882850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.882754 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5" Nov 02 20:30:11.901932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.901915 5800 scope.go:122] "RemoveContainer" containerID="b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024" Nov 02 20:30:11.902158 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:11.902138 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024\": container with ID starting with b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024 not found: ID does not exist" containerID="b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024" Nov 02 20:30:11.902211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.902163 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024"} err="failed to get container status \"b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024\": rpc error: code = NotFound desc = could not find container \"b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024\": container with ID starting with b6e90472b72248de57eaef16ba95a7af51a918ac270757a3019f16500a84d024 not found: ID does not exist" Nov 02 20:30:11.902211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.902179 5800 scope.go:122] "RemoveContainer" containerID="fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380" Nov 02 20:30:11.913935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.913915 5800 scope.go:122] "RemoveContainer" containerID="fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380" Nov 02 20:30:11.914208 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:11.914190 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380\": container with ID starting with fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380 not found: ID does not exist" containerID="fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380" Nov 02 20:30:11.914271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:11.914214 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380"} err="failed to get container status \"fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380\": rpc error: code = NotFound desc = could not find container \"fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380\": container with ID starting with fb2bb3f6908cf5b32740bdfca135db5b1aa6cb183e29c2d4793b113a1c4a5380 not found: ID does not exist" Nov 02 20:30:12.064390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:12.064044 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5"] Nov 02 20:30:12.068187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:12.068154 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1bbd80-76c8c74f6c-zdln5"] Nov 02 20:30:12.160303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:12.160238 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c"] Nov 02 20:30:12.163183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:12.163159 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1bbd80-7cd7787bc9-d7g8c"] Nov 02 20:30:13.625277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:13.625246 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25aaa18e-437d-4149-a284-e06b174ab477" path="/var/lib/kubelet/pods/25aaa18e-437d-4149-a284-e06b174ab477/volumes" Nov 02 20:30:13.641305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:13.625908 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c504e0-833e-41f5-899a-1fb20e1d2a50" path="/var/lib/kubelet/pods/e3c504e0-833e-41f5-899a-1fb20e1d2a50/volumes" Nov 02 20:30:13.858670 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:13.858639 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr"] Nov 02 20:30:14.250355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.249562 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:14.312505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.312472 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03c94f8-a8f9-464e-a350-b5e35da9f839-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03c94f8-a8f9-464e-a350-b5e35da9f839-maas-api-service-ca\") pod \"d03c94f8-a8f9-464e-a350-b5e35da9f839\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " Nov 02 20:30:14.312688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.312570 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03c94f8-a8f9-464e-a350-b5e35da9f839-kube-api-access-lmzl7\" (UniqueName: \"kubernetes.io/projected/d03c94f8-a8f9-464e-a350-b5e35da9f839-kube-api-access-lmzl7\") pod \"d03c94f8-a8f9-464e-a350-b5e35da9f839\" (UID: \"d03c94f8-a8f9-464e-a350-b5e35da9f839\") " Nov 02 20:30:14.312820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.312797 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03c94f8-a8f9-464e-a350-b5e35da9f839-maas-api-service-ca" pod "d03c94f8-a8f9-464e-a350-b5e35da9f839" (UID: "d03c94f8-a8f9-464e-a350-b5e35da9f839"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:14.313026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.313013 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03c94f8-a8f9-464e-a350-b5e35da9f839-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:14.314676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.314653 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03c94f8-a8f9-464e-a350-b5e35da9f839-kube-api-access-lmzl7" pod "d03c94f8-a8f9-464e-a350-b5e35da9f839" (UID: "d03c94f8-a8f9-464e-a350-b5e35da9f839"). InnerVolumeSpecName "kube-api-access-lmzl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:14.414464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.414424 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmzl7\" (UniqueName: \"kubernetes.io/projected/d03c94f8-a8f9-464e-a350-b5e35da9f839-kube-api-access-lmzl7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:14.918331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.918299 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" Nov 02 20:30:14.918331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.918314 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr" event={"ID":"d03c94f8-a8f9-464e-a350-b5e35da9f839","Type":"ContainerDied","Data":"63273a0f7c4d44ab1e7b420065a8a367e51b61c088109224ef4a7155471e5c33"} Nov 02 20:30:14.918913 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:14.918385 5800 scope.go:122] "RemoveContainer" containerID="dea7b71d537197c4264584058d2d82ed16b8a093ee402e6d230ed0490cc5c679" Nov 02 20:30:15.017687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.017653 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m"] Nov 02 20:30:15.088090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.088038 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr"] Nov 02 20:30:15.098601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.098563 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-1bbd80-29894190-6l5fr"] Nov 02 20:30:15.303420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.303385 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:15.325320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.322846 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-maas-api-service-ca\") pod \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " Nov 02 20:30:15.325320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.322973 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-kube-api-access-76fs6\" (UniqueName: \"kubernetes.io/projected/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-kube-api-access-76fs6\") pod \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\" (UID: \"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c\") " Nov 02 20:30:15.325320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.324256 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-maas-api-service-ca" pod "8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" (UID: "8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:15.395310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.391517 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-kube-api-access-76fs6" pod "8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" (UID: "8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c"). InnerVolumeSpecName "kube-api-access-76fs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:15.435303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.427301 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:15.435303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.427362 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fs6\" (UniqueName: \"kubernetes.io/projected/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c-kube-api-access-76fs6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:15.614444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.614408 5800 scope.go:122] "RemoveContainer" containerID="94848f62a268262946d228771545353c417698cfcbd7bc592aa732cedbab4f82" Nov 02 20:30:15.624072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.624044 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" path="/var/lib/kubelet/pods/d03c94f8-a8f9-464e-a350-b5e35da9f839/volumes" Nov 02 20:30:15.928622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.928532 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" event={"ID":"8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c","Type":"ContainerDied","Data":"95cd51c4bf0a493226164f3d7d29cd346d5363f71673a7d992d6bd551006bf92"} Nov 02 20:30:15.928622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.928612 5800 scope.go:122] "RemoveContainer" containerID="051a59f3bca26ae9edead31cc81c7aa5f2ca041aed1b6c3a2170083767ef8cc1" Nov 02 20:30:15.929184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:15.928707 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m" Nov 02 20:30:16.083682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:16.083647 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m"] Nov 02 20:30:16.089799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:16.089769 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-b028e6-29894190-cfc7m"] Nov 02 20:30:16.614336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:16.614302 5800 scope.go:122] "RemoveContainer" containerID="d128fd3de319afd3ad3a19b9fec08a5d61e8b3091f41f7030a2e30e639d1a059" Nov 02 20:30:16.942259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:16.942168 5800 generic.go:356] "Generic (PLEG): container finished" podID="401e8925-3666-479e-bcb1-770823db1c43" containerID="2bf05110db35d6f41ae3c3c60dbf87f686b0a63cd0b71279aac42e5ac41fb2bb" exitCode=60 Nov 02 20:30:16.942259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:16.942232 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" event={"ID":"401e8925-3666-479e-bcb1-770823db1c43","Type":"ContainerDied","Data":"2bf05110db35d6f41ae3c3c60dbf87f686b0a63cd0b71279aac42e5ac41fb2bb"} Nov 02 20:30:16.942833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:16.942321 5800 scope.go:122] "RemoveContainer" containerID="94848f62a268262946d228771545353c417698cfcbd7bc592aa732cedbab4f82" Nov 02 20:30:16.943505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:16.943114 5800 scope.go:122] "RemoveContainer" containerID="2bf05110db35d6f41ae3c3c60dbf87f686b0a63cd0b71279aac42e5ac41fb2bb" Nov 02 20:30:16.943640 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:16.943568 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc_odh-ai-gateway-infra(401e8925-3666-479e-bcb1-770823db1c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" podUID="401e8925-3666-479e-bcb1-770823db1c43" Nov 02 20:30:17.628370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:17.626941 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" path="/var/lib/kubelet/pods/8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c/volumes" Nov 02 20:30:17.629814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:17.629408 5800 scope.go:122] "RemoveContainer" containerID="8e8ba78db0b9088f4987d276f4c879c69bdb1fec21617b427e6dd8eea64baf8c" Nov 02 20:30:17.631512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:17.630322 5800 scope.go:122] "RemoveContainer" containerID="aa4febc464401943dcdc0ae1322e79432da0111249367c0c6eb6b112a247f04e" Nov 02 20:30:17.956533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:17.956499 5800 generic.go:356] "Generic (PLEG): container finished" podID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerID="1454dba3e0cfeb96252d8eef13097958298d42c65abd69aea627b4d1cf5d337f" exitCode=60 Nov 02 20:30:17.956979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:17.956575 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" event={"ID":"3ae7e43b-a128-45e0-b2e5-f32b4350d81c","Type":"ContainerDied","Data":"1454dba3e0cfeb96252d8eef13097958298d42c65abd69aea627b4d1cf5d337f"} Nov 02 20:30:17.956979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:17.956657 5800 scope.go:122] "RemoveContainer" containerID="d128fd3de319afd3ad3a19b9fec08a5d61e8b3091f41f7030a2e30e639d1a059" Nov 02 20:30:17.957117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:17.957104 5800 scope.go:122] "RemoveContainer" containerID="1454dba3e0cfeb96252d8eef13097958298d42c65abd69aea627b4d1cf5d337f" Nov 02 20:30:17.957743 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:17.957502 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl_odh-ai-gateway-infra(3ae7e43b-a128-45e0-b2e5-f32b4350d81c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" Nov 02 20:30:18.064527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.064494 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc"] Nov 02 20:30:18.307050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.307022 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:18.418796 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.418765 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401e8925-3666-479e-bcb1-770823db1c43-kube-api-access-gvn9d\" (UniqueName: \"kubernetes.io/projected/401e8925-3666-479e-bcb1-770823db1c43-kube-api-access-gvn9d\") pod \"401e8925-3666-479e-bcb1-770823db1c43\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " Nov 02 20:30:18.418970 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.418853 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401e8925-3666-479e-bcb1-770823db1c43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e8925-3666-479e-bcb1-770823db1c43-maas-api-service-ca\") pod \"401e8925-3666-479e-bcb1-770823db1c43\" (UID: \"401e8925-3666-479e-bcb1-770823db1c43\") " Nov 02 20:30:18.419314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.419261 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401e8925-3666-479e-bcb1-770823db1c43-maas-api-service-ca" pod "401e8925-3666-479e-bcb1-770823db1c43" (UID: "401e8925-3666-479e-bcb1-770823db1c43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:18.439092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.439047 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401e8925-3666-479e-bcb1-770823db1c43-kube-api-access-gvn9d" pod "401e8925-3666-479e-bcb1-770823db1c43" (UID: "401e8925-3666-479e-bcb1-770823db1c43"). InnerVolumeSpecName "kube-api-access-gvn9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:18.520801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.520759 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvn9d\" (UniqueName: \"kubernetes.io/projected/401e8925-3666-479e-bcb1-770823db1c43-kube-api-access-gvn9d\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:18.520801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.520795 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/401e8925-3666-479e-bcb1-770823db1c43-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:18.614600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.614518 5800 scope.go:122] "RemoveContainer" containerID="541a3fd4dd2bc71d3f15be36d4e116ce36a5980cc1f7c1e41135b227d3efb38a" Nov 02 20:30:18.684676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.684627 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw"] Nov 02 20:30:18.684932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.684898 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" containerID="cri-o://74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8" gracePeriod=30 Nov 02 20:30:18.969119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.969083 5800 generic.go:356] "Generic (PLEG): container finished" podID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerID="15f18a44b2f562d94d48c0cc88b16215bd57062f83f596354183dee8371da30f" exitCode=60 Nov 02 20:30:18.969609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.969160 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" event={"ID":"f4561427-fd5d-4a8f-bacd-fd1b46c92778","Type":"ContainerDied","Data":"15f18a44b2f562d94d48c0cc88b16215bd57062f83f596354183dee8371da30f"} Nov 02 20:30:18.969609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.969237 5800 scope.go:122] "RemoveContainer" containerID="8e8ba78db0b9088f4987d276f4c879c69bdb1fec21617b427e6dd8eea64baf8c" Nov 02 20:30:18.969734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.969629 5800 scope.go:122] "RemoveContainer" containerID="15f18a44b2f562d94d48c0cc88b16215bd57062f83f596354183dee8371da30f" Nov 02 20:30:18.969936 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:18.969914 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894190-5cs67_odh-ai-gateway-infra(f4561427-fd5d-4a8f-bacd-fd1b46c92778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" Nov 02 20:30:18.971303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.971273 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" Nov 02 20:30:18.971409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.971307 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc" event={"ID":"401e8925-3666-479e-bcb1-770823db1c43","Type":"ContainerDied","Data":"0899d3719844de17977cab5304e1f0b1844641d6c4d6c51ee8f2226f1885586d"} Nov 02 20:30:18.973813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.973788 5800 generic.go:356] "Generic (PLEG): container finished" podID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerID="42b63266851bf79d7a4881059176a3aec37bfbbdd652c64dafac0300edcd13d6" exitCode=60 Nov 02 20:30:18.973930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.973826 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" event={"ID":"f004bee4-4b80-47f3-8468-bd82fcdd12e5","Type":"ContainerDied","Data":"42b63266851bf79d7a4881059176a3aec37bfbbdd652c64dafac0300edcd13d6"} Nov 02 20:30:18.974325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.974256 5800 scope.go:122] "RemoveContainer" containerID="42b63266851bf79d7a4881059176a3aec37bfbbdd652c64dafac0300edcd13d6" Nov 02 20:30:18.974597 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:18.974570 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf_odh-ai-gateway-infra(f004bee4-4b80-47f3-8468-bd82fcdd12e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" Nov 02 20:30:18.984534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.984513 5800 scope.go:122] "RemoveContainer" containerID="2bf05110db35d6f41ae3c3c60dbf87f686b0a63cd0b71279aac42e5ac41fb2bb" Nov 02 20:30:18.996122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:18.996096 5800 scope.go:122] "RemoveContainer" containerID="aa4febc464401943dcdc0ae1322e79432da0111249367c0c6eb6b112a247f04e" Nov 02 20:30:19.087363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.087328 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl"] Nov 02 20:30:19.183727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.183674 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc"] Nov 02 20:30:19.190321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.188056 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894190-l88kc"] Nov 02 20:30:19.291524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.291489 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:19.434336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.434216 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-maas-api-service-ca\") pod \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " Nov 02 20:30:19.434520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.434403 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-kube-api-access-zxncr\" (UniqueName: \"kubernetes.io/projected/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-kube-api-access-zxncr\") pod \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\" (UID: \"3ae7e43b-a128-45e0-b2e5-f32b4350d81c\") " Nov 02 20:30:19.434673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.434642 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-maas-api-service-ca" pod "3ae7e43b-a128-45e0-b2e5-f32b4350d81c" (UID: "3ae7e43b-a128-45e0-b2e5-f32b4350d81c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:19.434948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.434927 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:19.436460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.436439 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-kube-api-access-zxncr" pod "3ae7e43b-a128-45e0-b2e5-f32b4350d81c" (UID: "3ae7e43b-a128-45e0-b2e5-f32b4350d81c"). InnerVolumeSpecName "kube-api-access-zxncr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:19.536646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.536606 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxncr\" (UniqueName: \"kubernetes.io/projected/3ae7e43b-a128-45e0-b2e5-f32b4350d81c-kube-api-access-zxncr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:19.623845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:19.623812 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401e8925-3666-479e-bcb1-770823db1c43" path="/var/lib/kubelet/pods/401e8925-3666-479e-bcb1-770823db1c43/volumes" Nov 02 20:30:20.005446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.005410 5800 generic.go:356] "Generic (PLEG): container finished" podID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerID="78c0b0a20663ce2154c9b636889da7d757d4e0952a9443997b4aef12c9ba5d4c" exitCode=60 Nov 02 20:30:20.005919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.005526 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" event={"ID":"bba21cfc-1699-4d9a-90ee-a40b94edbc80","Type":"ContainerDied","Data":"78c0b0a20663ce2154c9b636889da7d757d4e0952a9443997b4aef12c9ba5d4c"} Nov 02 20:30:20.005919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.005569 5800 scope.go:122] "RemoveContainer" containerID="541a3fd4dd2bc71d3f15be36d4e116ce36a5980cc1f7c1e41135b227d3efb38a" Nov 02 20:30:20.005919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.005916 5800 scope.go:122] "RemoveContainer" containerID="78c0b0a20663ce2154c9b636889da7d757d4e0952a9443997b4aef12c9ba5d4c" Nov 02 20:30:20.006381 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:20.006346 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4_odh-ai-gateway-infra(bba21cfc-1699-4d9a-90ee-a40b94edbc80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" Nov 02 20:30:20.008864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.008842 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" Nov 02 20:30:20.008995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.008879 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl" event={"ID":"3ae7e43b-a128-45e0-b2e5-f32b4350d81c","Type":"ContainerDied","Data":"497b405bbc5dbcbf3210ccbc9feeeed85dc04b16e702bfb93b8856852ec0e0a4"} Nov 02 20:30:20.011075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.011048 5800 generic.go:356] "Generic (PLEG): container finished" podID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerID="b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0" exitCode=0 Nov 02 20:30:20.011248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.011157 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" event={"ID":"2ba9d419-bf87-4c95-9932-1954286e1a08","Type":"ContainerDied","Data":"b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0"} Nov 02 20:30:20.011600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.011582 5800 scope.go:122] "RemoveContainer" containerID="b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0" Nov 02 20:30:20.013331 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.013298 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:20.013436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.013384 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:20.013510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.013437 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:20.021751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.021731 5800 scope.go:122] "RemoveContainer" containerID="1454dba3e0cfeb96252d8eef13097958298d42c65abd69aea627b4d1cf5d337f" Nov 02 20:30:20.105591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.105455 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67"] Nov 02 20:30:20.286780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.286748 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:20.299436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.299405 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf"] Nov 02 20:30:20.453409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.453373 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4561427-fd5d-4a8f-bacd-fd1b46c92778-kube-api-access-f426q\" (UniqueName: \"kubernetes.io/projected/f4561427-fd5d-4a8f-bacd-fd1b46c92778-kube-api-access-f426q\") pod \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " Nov 02 20:30:20.453699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.453474 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4561427-fd5d-4a8f-bacd-fd1b46c92778-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4561427-fd5d-4a8f-bacd-fd1b46c92778-maas-api-service-ca\") pod \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\" (UID: \"f4561427-fd5d-4a8f-bacd-fd1b46c92778\") " Nov 02 20:30:20.454050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.454014 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4561427-fd5d-4a8f-bacd-fd1b46c92778-maas-api-service-ca" pod "f4561427-fd5d-4a8f-bacd-fd1b46c92778" (UID: "f4561427-fd5d-4a8f-bacd-fd1b46c92778"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:20.454556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.454530 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl"] Nov 02 20:30:20.456162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.456138 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4561427-fd5d-4a8f-bacd-fd1b46c92778-kube-api-access-f426q" pod "f4561427-fd5d-4a8f-bacd-fd1b46c92778" (UID: "f4561427-fd5d-4a8f-bacd-fd1b46c92778"). InnerVolumeSpecName "kube-api-access-f426q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:20.464691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.464667 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-7cad76-29894190-x9rxl"] Nov 02 20:30:20.480366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.480344 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:20.555154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.555069 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f426q\" (UniqueName: \"kubernetes.io/projected/f4561427-fd5d-4a8f-bacd-fd1b46c92778-kube-api-access-f426q\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.555154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.555101 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4561427-fd5d-4a8f-bacd-fd1b46c92778-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.655490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.655460 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f004bee4-4b80-47f3-8468-bd82fcdd12e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f004bee4-4b80-47f3-8468-bd82fcdd12e5-maas-api-service-ca\") pod \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " Nov 02 20:30:20.655673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.655583 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004bee4-4b80-47f3-8468-bd82fcdd12e5-kube-api-access-7qllk\" (UniqueName: \"kubernetes.io/projected/f004bee4-4b80-47f3-8468-bd82fcdd12e5-kube-api-access-7qllk\") pod \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\" (UID: \"f004bee4-4b80-47f3-8468-bd82fcdd12e5\") " Nov 02 20:30:20.656339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.656311 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f004bee4-4b80-47f3-8468-bd82fcdd12e5-maas-api-service-ca" pod "f004bee4-4b80-47f3-8468-bd82fcdd12e5" (UID: "f004bee4-4b80-47f3-8468-bd82fcdd12e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:20.658804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.658779 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004bee4-4b80-47f3-8468-bd82fcdd12e5-kube-api-access-7qllk" pod "f004bee4-4b80-47f3-8468-bd82fcdd12e5" (UID: "f004bee4-4b80-47f3-8468-bd82fcdd12e5"). InnerVolumeSpecName "kube-api-access-7qllk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:20.757377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.757332 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f004bee4-4b80-47f3-8468-bd82fcdd12e5-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:20.757377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:20.757370 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qllk\" (UniqueName: \"kubernetes.io/projected/f004bee4-4b80-47f3-8468-bd82fcdd12e5-kube-api-access-7qllk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.020566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.020537 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" Nov 02 20:30:21.020997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.020537 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf" event={"ID":"f004bee4-4b80-47f3-8468-bd82fcdd12e5","Type":"ContainerDied","Data":"34100d9913934851f87809f872977fe67a30005fd85b62e9b5c66702088cbffa"} Nov 02 20:30:21.020997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.020682 5800 scope.go:122] "RemoveContainer" containerID="42b63266851bf79d7a4881059176a3aec37bfbbdd652c64dafac0300edcd13d6" Nov 02 20:30:21.054807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.054081 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" event={"ID":"2ba9d419-bf87-4c95-9932-1954286e1a08","Type":"ContainerStarted","Data":"f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664"} Nov 02 20:30:21.056232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.056205 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" event={"ID":"f4561427-fd5d-4a8f-bacd-fd1b46c92778","Type":"ContainerDied","Data":"f25185d3489450a001d26e39f94d04c252f8b29135f48623cc91121560fdd21f"} Nov 02 20:30:21.056408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.056271 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67" Nov 02 20:30:21.060267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.060239 5800 scope.go:122] "RemoveContainer" containerID="15f18a44b2f562d94d48c0cc88b16215bd57062f83f596354183dee8371da30f" Nov 02 20:30:21.122944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.122870 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4"] Nov 02 20:30:21.175934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.175898 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67"] Nov 02 20:30:21.181863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.181827 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894190-5cs67"] Nov 02 20:30:21.265640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.265598 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:30:21.265840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.265665 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:30:21.349306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.333029 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf"] Nov 02 20:30:21.357045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.356907 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894190-8kfbf"] Nov 02 20:30:21.361896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.361870 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:21.469742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.469663 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba21cfc-1699-4d9a-90ee-a40b94edbc80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba21cfc-1699-4d9a-90ee-a40b94edbc80-maas-api-service-ca\") pod \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " Nov 02 20:30:21.469918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.469748 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba21cfc-1699-4d9a-90ee-a40b94edbc80-kube-api-access-vn9zp\" (UniqueName: \"kubernetes.io/projected/bba21cfc-1699-4d9a-90ee-a40b94edbc80-kube-api-access-vn9zp\") pod \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\" (UID: \"bba21cfc-1699-4d9a-90ee-a40b94edbc80\") " Nov 02 20:30:21.470111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.470081 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba21cfc-1699-4d9a-90ee-a40b94edbc80-maas-api-service-ca" pod "bba21cfc-1699-4d9a-90ee-a40b94edbc80" (UID: "bba21cfc-1699-4d9a-90ee-a40b94edbc80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:21.470277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.470262 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba21cfc-1699-4d9a-90ee-a40b94edbc80-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.471951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.471924 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba21cfc-1699-4d9a-90ee-a40b94edbc80-kube-api-access-vn9zp" pod "bba21cfc-1699-4d9a-90ee-a40b94edbc80" (UID: "bba21cfc-1699-4d9a-90ee-a40b94edbc80"). InnerVolumeSpecName "kube-api-access-vn9zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:21.571674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.571642 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn9zp\" (UniqueName: \"kubernetes.io/projected/bba21cfc-1699-4d9a-90ee-a40b94edbc80-kube-api-access-vn9zp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.651754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.651722 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" path="/var/lib/kubelet/pods/3ae7e43b-a128-45e0-b2e5-f32b4350d81c/volumes" Nov 02 20:30:21.652188 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.652171 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" path="/var/lib/kubelet/pods/f004bee4-4b80-47f3-8468-bd82fcdd12e5/volumes" Nov 02 20:30:21.652791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.652766 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" path="/var/lib/kubelet/pods/f4561427-fd5d-4a8f-bacd-fd1b46c92778/volumes" Nov 02 20:30:21.712756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.712721 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:30:21.714226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.714199 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 02 20:30:21.714393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:21.714253 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 02 20:30:22.075121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:22.074081 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" Nov 02 20:30:22.075121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:22.074114 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4" event={"ID":"bba21cfc-1699-4d9a-90ee-a40b94edbc80","Type":"ContainerDied","Data":"1dfbf0b17898c40c3cf8a43f7258415756a675111aeee2ab3e5934ba8d9433cf"} Nov 02 20:30:22.075121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:22.074193 5800 scope.go:122] "RemoveContainer" containerID="78c0b0a20663ce2154c9b636889da7d757d4e0952a9443997b4aef12c9ba5d4c" Nov 02 20:30:22.248380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:22.248337 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4"] Nov 02 20:30:22.250846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:22.250815 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-cdbffd-29894190-bqww4"] Nov 02 20:30:22.697172 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:22.697137 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:30:22.712579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:22.712549 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 02 20:30:22.712736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:22.712598 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 02 20:30:23.624140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:23.624106 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" path="/var/lib/kubelet/pods/bba21cfc-1699-4d9a-90ee-a40b94edbc80/volumes" Nov 02 20:30:23.713385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:23.713275 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 02 20:30:23.713385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:23.713357 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 02 20:30:23.923668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:23.923644 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:30:23.959459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:23.959435 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:30:24.093561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.093525 5800 generic.go:356] "Generic (PLEG): container finished" podID="884d91e8-092f-4610-9d29-638a46aad29d" containerID="74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8" exitCode=0 Nov 02 20:30:24.093753 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.093595 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" Nov 02 20:30:24.093753 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.093614 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" event={"ID":"884d91e8-092f-4610-9d29-638a46aad29d","Type":"ContainerDied","Data":"74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8"} Nov 02 20:30:24.093753 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.093667 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw" event={"ID":"884d91e8-092f-4610-9d29-638a46aad29d","Type":"ContainerDied","Data":"20a0bf5213712df5e169af68d36eae2d13e48608b3b8e060a30e54bfa89f78d1"} Nov 02 20:30:24.093753 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.093691 5800 scope.go:122] "RemoveContainer" containerID="74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8" Nov 02 20:30:24.104882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.104859 5800 scope.go:122] "RemoveContainer" containerID="942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25" Nov 02 20:30:24.113552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113521 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-certs\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.113680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113577 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884d91e8-092f-4610-9d29-638a46aad29d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884d91e8-092f-4610-9d29-638a46aad29d-istiod-ca-cert\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.113680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113631 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-istio-token\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-istio-token\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.113680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113667 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/884d91e8-092f-4610-9d29-638a46aad29d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884d91e8-092f-4610-9d29-638a46aad29d-istio-podinfo\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.113991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113756 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-socket\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.113991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113797 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-certs" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:24.113991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113928 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-kube-api-access-9dsq5\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-kube-api-access-9dsq5\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.113991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.113975 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-credential-socket\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.114262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.114052 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-envoy\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.114262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.114067 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-socket" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:24.114262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.114138 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-data\") pod \"884d91e8-092f-4610-9d29-638a46aad29d\" (UID: \"884d91e8-092f-4610-9d29-638a46aad29d\") " Nov 02 20:30:24.114534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.114470 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884d91e8-092f-4610-9d29-638a46aad29d-istiod-ca-cert" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:24.114534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.114512 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-credential-socket" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:24.114714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.114690 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-data" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:24.115208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115184 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.115350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115213 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.115350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115227 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.115350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115241 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884d91e8-092f-4610-9d29-638a46aad29d-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.115350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115254 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.115688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115670 5800 scope.go:122] "RemoveContainer" containerID="74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8" Nov 02 20:30:24.115976 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:24.115955 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8\": container with ID starting with 74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8 not found: ID does not exist" containerID="74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8" Nov 02 20:30:24.116077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115979 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8"} err="failed to get container status \"74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8\": rpc error: code = NotFound desc = could not find container \"74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8\": container with ID starting with 74778606919bfacc4699fe3b4b714d6db005b18dde934f6ec5f0944d9bc56ac8 not found: ID does not exist" Nov 02 20:30:24.116077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.115996 5800 scope.go:122] "RemoveContainer" containerID="942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25" Nov 02 20:30:24.116292 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:24.116256 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25\": container with ID starting with 942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25 not found: ID does not exist" containerID="942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25" Nov 02 20:30:24.116394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.116296 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25"} err="failed to get container status \"942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25\": rpc error: code = NotFound desc = could not find container \"942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25\": container with ID starting with 942a47d176ed6ceac0a86851b991072ac5fe90a07239f0a82be802e72bfc2b25 not found: ID does not exist" Nov 02 20:30:24.117470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.117443 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-kube-api-access-9dsq5" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "kube-api-access-9dsq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:24.117578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.117492 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-istio-token" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:24.117762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.117725 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/884d91e8-092f-4610-9d29-638a46aad29d-istio-podinfo" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:24.118632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.118608 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-envoy" pod "884d91e8-092f-4610-9d29-638a46aad29d" (UID: "884d91e8-092f-4610-9d29-638a46aad29d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:24.216045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.216007 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dsq5\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-kube-api-access-9dsq5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.216045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.216040 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884d91e8-092f-4610-9d29-638a46aad29d-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.216045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.216053 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884d91e8-092f-4610-9d29-638a46aad29d-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.216351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.216062 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884d91e8-092f-4610-9d29-638a46aad29d-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:24.510336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.510257 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw"] Nov 02 20:30:24.515054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.515026 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1bbd80-openshift-default-58c5c4f769-xzdvw"] Nov 02 20:30:24.713375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.713262 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 02 20:30:24.713375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:24.713365 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 02 20:30:25.624905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:25.624871 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884d91e8-092f-4610-9d29-638a46aad29d" path="/var/lib/kubelet/pods/884d91e8-092f-4610-9d29-638a46aad29d/volumes" Nov 02 20:30:25.713128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:25.713084 5800 patch_prober.go:29] interesting pod/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 02 20:30:25.713352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:25.713157 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 02 20:30:26.717814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:26.717784 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:30:26.718919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:26.718893 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:30:27.243380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.243333 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r"] Nov 02 20:30:27.243564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.243532 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" podUID="740e2f52-591a-4712-8898-a14aa59dd2a9" containerName="payload-processing" containerID="cri-o://e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a" gracePeriod=30 Nov 02 20:30:27.298207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.298176 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g"] Nov 02 20:30:27.298404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.298383 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" podUID="db1276ca-35a0-4335-89ba-d44d3c06d9d5" containerName="payload-pre-processing" containerID="cri-o://bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee" gracePeriod=30 Nov 02 20:30:27.598015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.597971 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:30:27.637798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.637772 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:30:27.678053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678022 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740e2f52-591a-4712-8898-a14aa59dd2a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740e2f52-591a-4712-8898-a14aa59dd2a9-plugins-config-volume\") pod \"740e2f52-591a-4712-8898-a14aa59dd2a9\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " Nov 02 20:30:27.678216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678106 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740e2f52-591a-4712-8898-a14aa59dd2a9-kube-api-access-t9d6f\" (UniqueName: \"kubernetes.io/projected/740e2f52-591a-4712-8898-a14aa59dd2a9-kube-api-access-t9d6f\") pod \"740e2f52-591a-4712-8898-a14aa59dd2a9\" (UID: \"740e2f52-591a-4712-8898-a14aa59dd2a9\") " Nov 02 20:30:27.678365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678219 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1276ca-35a0-4335-89ba-d44d3c06d9d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1276ca-35a0-4335-89ba-d44d3c06d9d5-plugins-config-volume\") pod \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " Nov 02 20:30:27.678365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678343 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1276ca-35a0-4335-89ba-d44d3c06d9d5-kube-api-access-nbppq\" (UniqueName: \"kubernetes.io/projected/db1276ca-35a0-4335-89ba-d44d3c06d9d5-kube-api-access-nbppq\") pod \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\" (UID: \"db1276ca-35a0-4335-89ba-d44d3c06d9d5\") " Nov 02 20:30:27.678590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678561 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740e2f52-591a-4712-8898-a14aa59dd2a9-plugins-config-volume" pod "740e2f52-591a-4712-8898-a14aa59dd2a9" (UID: "740e2f52-591a-4712-8898-a14aa59dd2a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:27.678697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678615 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1276ca-35a0-4335-89ba-d44d3c06d9d5-plugins-config-volume" pod "db1276ca-35a0-4335-89ba-d44d3c06d9d5" (UID: "db1276ca-35a0-4335-89ba-d44d3c06d9d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:27.678867 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678846 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740e2f52-591a-4712-8898-a14aa59dd2a9-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:27.678947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.678875 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1276ca-35a0-4335-89ba-d44d3c06d9d5-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:27.680352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.680328 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740e2f52-591a-4712-8898-a14aa59dd2a9-kube-api-access-t9d6f" pod "740e2f52-591a-4712-8898-a14aa59dd2a9" (UID: "740e2f52-591a-4712-8898-a14aa59dd2a9"). InnerVolumeSpecName "kube-api-access-t9d6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:27.680506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.680485 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1276ca-35a0-4335-89ba-d44d3c06d9d5-kube-api-access-nbppq" pod "db1276ca-35a0-4335-89ba-d44d3c06d9d5" (UID: "db1276ca-35a0-4335-89ba-d44d3c06d9d5"). InnerVolumeSpecName "kube-api-access-nbppq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:27.805190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.797537 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9d6f\" (UniqueName: \"kubernetes.io/projected/740e2f52-591a-4712-8898-a14aa59dd2a9-kube-api-access-t9d6f\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:27.805190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:27.797570 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbppq\" (UniqueName: \"kubernetes.io/projected/db1276ca-35a0-4335-89ba-d44d3c06d9d5-kube-api-access-nbppq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:28.131916 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.131822 5800 generic.go:356] "Generic (PLEG): container finished" podID="db1276ca-35a0-4335-89ba-d44d3c06d9d5" containerID="bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee" exitCode=0 Nov 02 20:30:28.131916 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.131874 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" Nov 02 20:30:28.132151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.131909 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" event={"ID":"db1276ca-35a0-4335-89ba-d44d3c06d9d5","Type":"ContainerDied","Data":"bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee"} Nov 02 20:30:28.132151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.131969 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g" event={"ID":"db1276ca-35a0-4335-89ba-d44d3c06d9d5","Type":"ContainerDied","Data":"c52d008427bb034ab6ab62a46e64f10e798e1697149634d7432fcce7492df537"} Nov 02 20:30:28.132151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.131998 5800 scope.go:122] "RemoveContainer" containerID="bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee" Nov 02 20:30:28.133574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.133548 5800 generic.go:356] "Generic (PLEG): container finished" podID="740e2f52-591a-4712-8898-a14aa59dd2a9" containerID="e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a" exitCode=0 Nov 02 20:30:28.133689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.133586 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" Nov 02 20:30:28.133689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.133634 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" event={"ID":"740e2f52-591a-4712-8898-a14aa59dd2a9","Type":"ContainerDied","Data":"e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a"} Nov 02 20:30:28.133806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.133691 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" event={"ID":"740e2f52-591a-4712-8898-a14aa59dd2a9","Type":"ContainerDied","Data":"8a4e51cc8777a9acabe2947e621183fa19c78446764ccae5781726fe4c30b06e"} Nov 02 20:30:28.143214 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.143190 5800 scope.go:122] "RemoveContainer" containerID="bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee" Nov 02 20:30:28.143597 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:28.143569 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee\": container with ID starting with bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee not found: ID does not exist" containerID="bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee" Nov 02 20:30:28.143713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.143609 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee"} err="failed to get container status \"bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee\": rpc error: code = NotFound desc = could not find container \"bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee\": container with ID starting with bc6e7195d6730d5c8be44451d1008205e9b00097538a46b7f5e1bcac612aa8ee not found: ID does not exist" Nov 02 20:30:28.143713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.143635 5800 scope.go:122] "RemoveContainer" containerID="e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a" Nov 02 20:30:28.153822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.153800 5800 scope.go:122] "RemoveContainer" containerID="e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a" Nov 02 20:30:28.154149 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:28.154125 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a\": container with ID starting with e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a not found: ID does not exist" containerID="e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a" Nov 02 20:30:28.154215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.154160 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a"} err="failed to get container status \"e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a\": rpc error: code = NotFound desc = could not find container \"e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a\": container with ID starting with e28ed39a2dd16bfdc1e4eebd85ccd9981b9e9ef6085bc08566d6a2d3bcc6f15a not found: ID does not exist" Nov 02 20:30:28.289368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.289324 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r"] Nov 02 20:30:28.291347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.291324 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r"] Nov 02 20:30:28.390355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.390250 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g"] Nov 02 20:30:28.393274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.393239 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-408103-6594c78b8c-dsr7g"] Nov 02 20:30:28.578034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:28.577991 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-408103-55cf765fbd-hbw9r" podUID="740e2f52-591a-4712-8898-a14aa59dd2a9" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.190:9004: i/o timeout" Nov 02 20:30:29.418001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:29.417966 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz"] Nov 02 20:30:29.594275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:29.594229 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs"] Nov 02 20:30:29.622839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:29.622809 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740e2f52-591a-4712-8898-a14aa59dd2a9" path="/var/lib/kubelet/pods/740e2f52-591a-4712-8898-a14aa59dd2a9/volumes" Nov 02 20:30:29.623204 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:29.623191 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1276ca-35a0-4335-89ba-d44d3c06d9d5" path="/var/lib/kubelet/pods/db1276ca-35a0-4335-89ba-d44d3c06d9d5/volumes" Nov 02 20:30:30.587498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:30.587304 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m"] Nov 02 20:30:30.587970 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:30.587562 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" containerID="cri-o://0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383" gracePeriod=30 Nov 02 20:30:33.478409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.478369 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g"] Nov 02 20:30:33.479047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479028 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.479047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479047 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479058 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479063 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479080 5800 cpu_manager.go:414] "containerMap: removing container" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479086 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479094 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479099 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479107 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479111 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479119 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479124 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479131 5800 cpu_manager.go:414] "containerMap: removing container" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479136 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479145 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479152 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479161 5800 cpu_manager.go:414] "containerMap: removing container" podUID="25aaa18e-437d-4149-a284-e06b174ab477" containerName="payload-pre-processing" Nov 02 20:30:33.479178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479166 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25aaa18e-437d-4149-a284-e06b174ab477" containerName="payload-pre-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479180 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479202 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479210 5800 cpu_manager.go:414] "containerMap: removing container" podUID="db1276ca-35a0-4335-89ba-d44d3c06d9d5" containerName="payload-pre-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479216 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1276ca-35a0-4335-89ba-d44d3c06d9d5" containerName="payload-pre-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479232 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479237 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479244 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479248 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479255 5800 cpu_manager.go:414] "containerMap: removing container" podUID="740e2f52-591a-4712-8898-a14aa59dd2a9" containerName="payload-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479259 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740e2f52-591a-4712-8898-a14aa59dd2a9" containerName="payload-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479265 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479270 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479295 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479303 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479313 5800 cpu_manager.go:414] "containerMap: removing container" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479318 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479326 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e3c504e0-833e-41f5-899a-1fb20e1d2a50" containerName="payload-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479332 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c504e0-833e-41f5-899a-1fb20e1d2a50" containerName="payload-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479338 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479342 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479350 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479355 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479445 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="25aaa18e-437d-4149-a284-e06b174ab477" containerName="payload-pre-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479455 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479461 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1276ca-35a0-4335-89ba-d44d3c06d9d5" containerName="payload-pre-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479467 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479474 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479480 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479485 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="740e2f52-591a-4712-8898-a14aa59dd2a9" containerName="payload-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479490 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479498 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479502 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479508 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbfb1f2-5486-4b4c-8390-f2ff2a0c7c6c" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479514 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c504e0-833e-41f5-899a-1fb20e1d2a50" containerName="payload-processing" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479520 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479525 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479531 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479536 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479543 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479549 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479554 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03c94f8-a8f9-464e-a350-b5e35da9f839" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479559 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479565 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:33.479996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.479570 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:33.484586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.484563 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.534777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.534739 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g"] Nov 02 20:30:33.576607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.576482 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:30:33.576607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.576566 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:30:33.576840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.576688 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:30:33.581461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.581434 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:30:33.587180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.587154 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.587271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.587215 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5q9\" (UniqueName: \"kubernetes.io/projected/e2d5312a-39be-47fa-bd43-18372611fdf9-kube-api-access-2b5q9\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.587473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.587450 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.689026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.688988 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.689226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.689041 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5q9\" (UniqueName: \"kubernetes.io/projected/e2d5312a-39be-47fa-bd43-18372611fdf9-kube-api-access-2b5q9\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.689226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.689162 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.689776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.689751 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.691404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.691386 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.742525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.742441 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5q9\" (UniqueName: \"kubernetes.io/projected/e2d5312a-39be-47fa-bd43-18372611fdf9-kube-api-access-2b5q9\") pod \"maas-api-revoke-sub-16d5e9ae-cn64g\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:33.798820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:33.798786 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:34.174923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.174889 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g"] Nov 02 20:30:34.175171 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:34.175145 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d5312a_39be_47fa_bd43_18372611fdf9.slice/crio-e37ec1349e679f89a8a1aa614b9d89a95b95a16fff89611081131506c4ca3cde WatchSource:0}: Error finding container e37ec1349e679f89a8a1aa614b9d89a95b95a16fff89611081131506c4ca3cde: Status 404 returned error can't find the container with id e37ec1349e679f89a8a1aa614b9d89a95b95a16fff89611081131506c4ca3cde Nov 02 20:30:34.196862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.196832 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" event={"ID":"e2d5312a-39be-47fa-bd43-18372611fdf9","Type":"ContainerStarted","Data":"e37ec1349e679f89a8a1aa614b9d89a95b95a16fff89611081131506c4ca3cde"} Nov 02 20:30:34.419091 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.419044 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" podUID="0d493684-7763-4db0-83f2-3b4726401899" containerName="payload-processing" containerID="cri-o://57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b" gracePeriod=25 Nov 02 20:30:34.595365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.595326 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" podUID="71f9487c-af13-49ba-a9b2-82963885abdf" containerName="payload-pre-processing" containerID="cri-o://f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722" gracePeriod=25 Nov 02 20:30:34.725577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.725543 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:30:34.801798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.801760 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d493684-7763-4db0-83f2-3b4726401899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d493684-7763-4db0-83f2-3b4726401899-plugins-config-volume\") pod \"0d493684-7763-4db0-83f2-3b4726401899\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " Nov 02 20:30:34.801993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.801896 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d493684-7763-4db0-83f2-3b4726401899-kube-api-access-kc5hs\" (UniqueName: \"kubernetes.io/projected/0d493684-7763-4db0-83f2-3b4726401899-kube-api-access-kc5hs\") pod \"0d493684-7763-4db0-83f2-3b4726401899\" (UID: \"0d493684-7763-4db0-83f2-3b4726401899\") " Nov 02 20:30:34.802524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.802485 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d493684-7763-4db0-83f2-3b4726401899-plugins-config-volume" pod "0d493684-7763-4db0-83f2-3b4726401899" (UID: "0d493684-7763-4db0-83f2-3b4726401899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:34.822568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.822426 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d493684-7763-4db0-83f2-3b4726401899-kube-api-access-kc5hs" pod "0d493684-7763-4db0-83f2-3b4726401899" (UID: "0d493684-7763-4db0-83f2-3b4726401899"). InnerVolumeSpecName "kube-api-access-kc5hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:34.898946 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.898922 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:30:34.903172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.903152 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d493684-7763-4db0-83f2-3b4726401899-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:34.903231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:34.903174 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5hs\" (UniqueName: \"kubernetes.io/projected/0d493684-7763-4db0-83f2-3b4726401899-kube-api-access-kc5hs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:35.004160 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.004113 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f9487c-af13-49ba-a9b2-82963885abdf-kube-api-access-sxswz\" (UniqueName: \"kubernetes.io/projected/71f9487c-af13-49ba-a9b2-82963885abdf-kube-api-access-sxswz\") pod \"71f9487c-af13-49ba-a9b2-82963885abdf\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " Nov 02 20:30:35.004426 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.004402 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f9487c-af13-49ba-a9b2-82963885abdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f9487c-af13-49ba-a9b2-82963885abdf-plugins-config-volume\") pod \"71f9487c-af13-49ba-a9b2-82963885abdf\" (UID: \"71f9487c-af13-49ba-a9b2-82963885abdf\") " Nov 02 20:30:35.004892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.004779 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f9487c-af13-49ba-a9b2-82963885abdf-plugins-config-volume" pod "71f9487c-af13-49ba-a9b2-82963885abdf" (UID: "71f9487c-af13-49ba-a9b2-82963885abdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:35.005084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.005065 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f9487c-af13-49ba-a9b2-82963885abdf-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:35.006700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.006673 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f9487c-af13-49ba-a9b2-82963885abdf-kube-api-access-sxswz" pod "71f9487c-af13-49ba-a9b2-82963885abdf" (UID: "71f9487c-af13-49ba-a9b2-82963885abdf"). InnerVolumeSpecName "kube-api-access-sxswz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:35.106584 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.106551 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxswz\" (UniqueName: \"kubernetes.io/projected/71f9487c-af13-49ba-a9b2-82963885abdf-kube-api-access-sxswz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:35.204243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.204210 5800 generic.go:356] "Generic (PLEG): container finished" podID="71f9487c-af13-49ba-a9b2-82963885abdf" containerID="f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722" exitCode=0 Nov 02 20:30:35.204462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.204261 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" Nov 02 20:30:35.204462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.204327 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" event={"ID":"71f9487c-af13-49ba-a9b2-82963885abdf","Type":"ContainerDied","Data":"f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722"} Nov 02 20:30:35.204462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.204379 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs" event={"ID":"71f9487c-af13-49ba-a9b2-82963885abdf","Type":"ContainerDied","Data":"6935bb6ab5c65f6c76fe55e3ccb9f85120ba2563fa142d26dc73a7e131915049"} Nov 02 20:30:35.204462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.204409 5800 scope.go:122] "RemoveContainer" containerID="f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722" Nov 02 20:30:35.206140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.206114 5800 generic.go:356] "Generic (PLEG): container finished" podID="e2d5312a-39be-47fa-bd43-18372611fdf9" containerID="0644e530d4b1e589b2f885fc23979b29a91d686793956b51ba618e24b9c4b2bf" exitCode=0 Nov 02 20:30:35.206295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.206208 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" event={"ID":"e2d5312a-39be-47fa-bd43-18372611fdf9","Type":"ContainerDied","Data":"0644e530d4b1e589b2f885fc23979b29a91d686793956b51ba618e24b9c4b2bf"} Nov 02 20:30:35.207705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.207683 5800 generic.go:356] "Generic (PLEG): container finished" podID="0d493684-7763-4db0-83f2-3b4726401899" containerID="57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b" exitCode=0 Nov 02 20:30:35.207802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.207736 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" event={"ID":"0d493684-7763-4db0-83f2-3b4726401899","Type":"ContainerDied","Data":"57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b"} Nov 02 20:30:35.207802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.207773 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" Nov 02 20:30:35.207880 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.207773 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz" event={"ID":"0d493684-7763-4db0-83f2-3b4726401899","Type":"ContainerDied","Data":"dae67ededae40fa6266c42996c551cf0e1b0066ca485b1050ba750a6200875a3"} Nov 02 20:30:35.224253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.224227 5800 scope.go:122] "RemoveContainer" containerID="f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722" Nov 02 20:30:35.224956 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:35.224856 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722\": container with ID starting with f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722 not found: ID does not exist" containerID="f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722" Nov 02 20:30:35.225037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.224973 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722"} err="failed to get container status \"f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722\": rpc error: code = NotFound desc = could not find container \"f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722\": container with ID starting with f8d74bec4b6b5723cce81ed6d1697bca7764b49de698129c50843e5668e82722 not found: ID does not exist" Nov 02 20:30:35.225037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.224997 5800 scope.go:122] "RemoveContainer" containerID="57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b" Nov 02 20:30:35.239333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.239274 5800 scope.go:122] "RemoveContainer" containerID="57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b" Nov 02 20:30:35.240457 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:35.240420 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b\": container with ID starting with 57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b not found: ID does not exist" containerID="57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b" Nov 02 20:30:35.240553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.240463 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b"} err="failed to get container status \"57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b\": rpc error: code = NotFound desc = could not find container \"57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b\": container with ID starting with 57e3a3a1e03d2551df5ad79432654e6071e714fae41b4695fede712deb01067b not found: ID does not exist" Nov 02 20:30:35.358053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.357940 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz"] Nov 02 20:30:35.360860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.360831 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdbffd-f76d999bc-hh4lz"] Nov 02 20:30:35.569271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.569236 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs"] Nov 02 20:30:35.574242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.574217 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdbffd-5fb85546c-wvwcs"] Nov 02 20:30:35.623766 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.623701 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d493684-7763-4db0-83f2-3b4726401899" path="/var/lib/kubelet/pods/0d493684-7763-4db0-83f2-3b4726401899/volumes" Nov 02 20:30:35.624265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.624245 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f9487c-af13-49ba-a9b2-82963885abdf" path="/var/lib/kubelet/pods/71f9487c-af13-49ba-a9b2-82963885abdf/volumes" Nov 02 20:30:35.862893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.862867 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:30:35.916488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916406 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-certs\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.916488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916473 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-podinfo\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.916703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916559 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istiod-ca-cert\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.916703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916650 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-token\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-token\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.916830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916704 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-envoy\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.916830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916718 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-certs" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:35.916830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916819 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-credential-socket\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.917003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916851 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-data\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.917003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916898 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-socket\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.917243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.916932 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-kube-api-access-zfktq\") pod \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\" (UID: \"7e78e4c5-6892-4e97-9f64-8e0c6da82a17\") " Nov 02 20:30:35.917806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.917407 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-data" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:35.917806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.917727 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-socket" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:35.919557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.919468 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istiod-ca-cert" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:35.919660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.919554 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-credential-socket" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:35.921590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.921109 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:35.921590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.921169 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:35.921590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.921187 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:35.921590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.921264 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-envoy" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:30:35.921837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.921604 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-token" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:35.922521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.922498 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-kube-api-access-zfktq" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "kube-api-access-zfktq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:35.922663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:35.922600 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-podinfo" pod "7e78e4c5-6892-4e97-9f64-8e0c6da82a17" (UID: "7e78e4c5-6892-4e97-9f64-8e0c6da82a17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:30:36.022589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.022553 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.022589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.022583 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-kube-api-access-zfktq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.022801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.022598 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.022801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.022611 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.022801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.022620 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.022801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.022631 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e78e4c5-6892-4e97-9f64-8e0c6da82a17-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.216309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.216196 5800 generic.go:356] "Generic (PLEG): container finished" podID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerID="0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383" exitCode=0 Nov 02 20:30:36.216309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.216245 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" Nov 02 20:30:36.216309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.216273 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" event={"ID":"7e78e4c5-6892-4e97-9f64-8e0c6da82a17","Type":"ContainerDied","Data":"0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383"} Nov 02 20:30:36.216610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.216353 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m" event={"ID":"7e78e4c5-6892-4e97-9f64-8e0c6da82a17","Type":"ContainerDied","Data":"6f732686a5ebfeb8a124d5aa29d26d6cd2b4aeb0593064d699b02d1a0e9806c3"} Nov 02 20:30:36.216610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.216383 5800 scope.go:122] "RemoveContainer" containerID="0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383" Nov 02 20:30:36.230248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.230220 5800 scope.go:122] "RemoveContainer" containerID="a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b" Nov 02 20:30:36.242582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.242548 5800 scope.go:122] "RemoveContainer" containerID="0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383" Nov 02 20:30:36.242875 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:36.242846 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383\": container with ID starting with 0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383 not found: ID does not exist" containerID="0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383" Nov 02 20:30:36.242974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.242872 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383"} err="failed to get container status \"0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383\": rpc error: code = NotFound desc = could not find container \"0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383\": container with ID starting with 0027f630eabaee44ad2fc9fdcbc75c3a4dbfe9fe3de8b2841b91cf36a28c9383 not found: ID does not exist" Nov 02 20:30:36.242974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.242889 5800 scope.go:122] "RemoveContainer" containerID="a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b" Nov 02 20:30:36.243111 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:36.243090 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b\": container with ID starting with a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b not found: ID does not exist" containerID="a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b" Nov 02 20:30:36.243220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.243115 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b"} err="failed to get container status \"a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b\": rpc error: code = NotFound desc = could not find container \"a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b\": container with ID starting with a4f5555b7aefec90ebd5ae6c50a2e96e5769f16e316122bc5d6bb4d31d29e49b not found: ID does not exist" Nov 02 20:30:36.370102 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.370063 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:36.370357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.370335 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m"] Nov 02 20:30:36.416733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.416701 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-408103-openshift-default-7d47467fb9-hqx6m"] Nov 02 20:30:36.426309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.425786 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-openshift-ca\") pod \"e2d5312a-39be-47fa-bd43-18372611fdf9\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " Nov 02 20:30:36.426309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.425857 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-xks-ca\") pod \"e2d5312a-39be-47fa-bd43-18372611fdf9\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " Nov 02 20:30:36.426309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.426055 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d5312a-39be-47fa-bd43-18372611fdf9-kube-api-access-2b5q9\" (UniqueName: \"kubernetes.io/projected/e2d5312a-39be-47fa-bd43-18372611fdf9-kube-api-access-2b5q9\") pod \"e2d5312a-39be-47fa-bd43-18372611fdf9\" (UID: \"e2d5312a-39be-47fa-bd43-18372611fdf9\") " Nov 02 20:30:36.426309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.426181 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-openshift-ca" pod "e2d5312a-39be-47fa-bd43-18372611fdf9" (UID: "e2d5312a-39be-47fa-bd43-18372611fdf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:36.426621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.426497 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.428722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.428687 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-xks-ca" pod "e2d5312a-39be-47fa-bd43-18372611fdf9" (UID: "e2d5312a-39be-47fa-bd43-18372611fdf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:36.428960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.428933 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d5312a-39be-47fa-bd43-18372611fdf9-kube-api-access-2b5q9" pod "e2d5312a-39be-47fa-bd43-18372611fdf9" (UID: "e2d5312a-39be-47fa-bd43-18372611fdf9"). InnerVolumeSpecName "kube-api-access-2b5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:36.527432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.527393 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d5312a-39be-47fa-bd43-18372611fdf9-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:36.527432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:36.527426 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5q9\" (UniqueName: \"kubernetes.io/projected/e2d5312a-39be-47fa-bd43-18372611fdf9-kube-api-access-2b5q9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:37.226981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:37.226948 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" Nov 02 20:30:37.227439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:37.226981 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g" event={"ID":"e2d5312a-39be-47fa-bd43-18372611fdf9","Type":"ContainerDied","Data":"e37ec1349e679f89a8a1aa614b9d89a95b95a16fff89611081131506c4ca3cde"} Nov 02 20:30:37.227439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:37.227033 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37ec1349e679f89a8a1aa614b9d89a95b95a16fff89611081131506c4ca3cde" Nov 02 20:30:37.624217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:37.624183 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" path="/var/lib/kubelet/pods/7e78e4c5-6892-4e97-9f64-8e0c6da82a17/volumes" Nov 02 20:30:49.399083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:49.399042 5800 generic.go:356] "Generic (PLEG): container finished" podID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerID="012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f" exitCode=0 Nov 02 20:30:49.399640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:49.399093 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" event={"ID":"c337afa1-a2b2-47fc-accb-4e11c7251b46","Type":"ContainerDied","Data":"012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f"} Nov 02 20:30:49.399732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:49.399713 5800 scope.go:122] "RemoveContainer" containerID="012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f" Nov 02 20:30:49.401228 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:49.401194 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:49.401365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:49.401267 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:49.401365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:49.401320 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:30:50.407782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:50.407740 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" event={"ID":"c337afa1-a2b2-47fc-accb-4e11c7251b46","Type":"ContainerStarted","Data":"7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56"} Nov 02 20:30:50.977969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:50.977930 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:30:50.978158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:50.978128 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:30:50.978208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:50.978172 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:30:51.814870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.814832 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm"] Nov 02 20:30:51.815422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815406 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e2d5312a-39be-47fa-bd43-18372611fdf9" containerName="revoke-keys" Nov 02 20:30:51.815422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815422 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d5312a-39be-47fa-bd43-18372611fdf9" containerName="revoke-keys" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815439 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815445 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815453 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0d493684-7763-4db0-83f2-3b4726401899" containerName="payload-processing" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815472 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d493684-7763-4db0-83f2-3b4726401899" containerName="payload-processing" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815480 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815485 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004bee4-4b80-47f3-8468-bd82fcdd12e5" containerName="cleanup" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815494 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815498 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815505 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815509 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815517 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815521 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba21cfc-1699-4d9a-90ee-a40b94edbc80" containerName="cleanup" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815534 5800 cpu_manager.go:414] "containerMap: removing container" podUID="71f9487c-af13-49ba-a9b2-82963885abdf" containerName="payload-pre-processing" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815539 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f9487c-af13-49ba-a9b2-82963885abdf" containerName="payload-pre-processing" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815548 5800 cpu_manager.go:414] "containerMap: removing container" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815552 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884d91e8-092f-4610-9d29-638a46aad29d" containerName="istio-proxy" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815563 5800 cpu_manager.go:414] "containerMap: removing container" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:51.815557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815568 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815662 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae7e43b-a128-45e0-b2e5-f32b4350d81c" containerName="cleanup" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815675 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="401e8925-3666-479e-bcb1-770823db1c43" containerName="cleanup" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815682 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d5312a-39be-47fa-bd43-18372611fdf9" containerName="revoke-keys" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815688 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4561427-fd5d-4a8f-bacd-fd1b46c92778" containerName="cleanup" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815694 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d493684-7763-4db0-83f2-3b4726401899" containerName="payload-processing" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815702 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815708 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e78e4c5-6892-4e97-9f64-8e0c6da82a17" containerName="istio-proxy" Nov 02 20:30:51.816531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.815715 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f9487c-af13-49ba-a9b2-82963885abdf" containerName="payload-pre-processing" Nov 02 20:30:51.819222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.819202 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.872327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.869703 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.872327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.869814 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlfm\" (UniqueName: \"kubernetes.io/projected/6acab08b-2437-4f42-8c45-f706f4d4afec-kube-api-access-2tlfm\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.872327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.869931 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.872327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.871749 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm"] Nov 02 20:30:51.907861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.907823 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:30:51.914127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.914105 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:30:51.914237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.914181 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:30:51.914294 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.914257 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:30:51.970646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.970612 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlfm\" (UniqueName: \"kubernetes.io/projected/6acab08b-2437-4f42-8c45-f706f4d4afec-kube-api-access-2tlfm\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.970840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.970707 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.970840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.970769 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.971552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.971525 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.973262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.973237 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:51.978462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.978436 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:30:51.978543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:51.978492 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:30:52.023929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:52.023899 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlfm\" (UniqueName: \"kubernetes.io/projected/6acab08b-2437-4f42-8c45-f706f4d4afec-kube-api-access-2tlfm\") pod \"maas-api-revoke-sub-3b7d3cca-ntqzm\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:52.161109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:52.161016 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:52.554996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:52.554962 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm"] Nov 02 20:30:52.557768 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:30:52.557731 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acab08b_2437_4f42_8c45_f706f4d4afec.slice/crio-af3f5ece5f53e5c1f855d01e730d5a971d68199bf432ccffa90a7abeef17509e WatchSource:0}: Error finding container af3f5ece5f53e5c1f855d01e730d5a971d68199bf432ccffa90a7abeef17509e: Status 404 returned error can't find the container with id af3f5ece5f53e5c1f855d01e730d5a971d68199bf432ccffa90a7abeef17509e Nov 02 20:30:52.774063 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:30:52.774026 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:30:52.977674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:52.977626 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:30:52.978104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:52.977694 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:30:53.435613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:53.435573 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" event={"ID":"6acab08b-2437-4f42-8c45-f706f4d4afec","Type":"ContainerStarted","Data":"5dbb6f3750fb92977cb2590abb0acfbd2cde4a2de8bf6bfc9e58910b490760bc"} Nov 02 20:30:53.436895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:53.436867 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" event={"ID":"6acab08b-2437-4f42-8c45-f706f4d4afec","Type":"ContainerStarted","Data":"af3f5ece5f53e5c1f855d01e730d5a971d68199bf432ccffa90a7abeef17509e"} Nov 02 20:30:53.545336 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:53.545250 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" podStartSLOduration=2.545226265 podStartE2EDuration="2.545226265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:53.543695981 +0000 UTC m=+2552.060468567" watchObservedRunningTime="2026-11-02 20:30:53.545226265 +0000 UTC m=+2552.061998851" Nov 02 20:30:53.978212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:53.978176 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:30:53.978658 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:53.978231 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:30:54.460303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:54.459402 5800 generic.go:356] "Generic (PLEG): container finished" podID="6acab08b-2437-4f42-8c45-f706f4d4afec" containerID="5dbb6f3750fb92977cb2590abb0acfbd2cde4a2de8bf6bfc9e58910b490760bc" exitCode=0 Nov 02 20:30:54.460303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:54.459585 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" event={"ID":"6acab08b-2437-4f42-8c45-f706f4d4afec","Type":"ContainerDied","Data":"5dbb6f3750fb92977cb2590abb0acfbd2cde4a2de8bf6bfc9e58910b490760bc"} Nov 02 20:30:54.978497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:54.978455 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 02 20:30:54.979005 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:54.978509 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 02 20:30:55.795532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.788050 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:55.830896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.830857 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6acab08b-2437-4f42-8c45-f706f4d4afec-kube-api-access-2tlfm\" (UniqueName: \"kubernetes.io/projected/6acab08b-2437-4f42-8c45-f706f4d4afec-kube-api-access-2tlfm\") pod \"6acab08b-2437-4f42-8c45-f706f4d4afec\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " Nov 02 20:30:55.831067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.830926 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-openshift-ca\") pod \"6acab08b-2437-4f42-8c45-f706f4d4afec\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " Nov 02 20:30:55.831067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.831039 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-xks-ca\") pod \"6acab08b-2437-4f42-8c45-f706f4d4afec\" (UID: \"6acab08b-2437-4f42-8c45-f706f4d4afec\") " Nov 02 20:30:55.831784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.831747 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-openshift-ca" pod "6acab08b-2437-4f42-8c45-f706f4d4afec" (UID: "6acab08b-2437-4f42-8c45-f706f4d4afec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:55.834091 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.834049 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-xks-ca" pod "6acab08b-2437-4f42-8c45-f706f4d4afec" (UID: "6acab08b-2437-4f42-8c45-f706f4d4afec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:30:55.834212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.834137 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acab08b-2437-4f42-8c45-f706f4d4afec-kube-api-access-2tlfm" pod "6acab08b-2437-4f42-8c45-f706f4d4afec" (UID: "6acab08b-2437-4f42-8c45-f706f4d4afec"). InnerVolumeSpecName "kube-api-access-2tlfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:55.933062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.933014 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:55.933062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.933054 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlfm\" (UniqueName: \"kubernetes.io/projected/6acab08b-2437-4f42-8c45-f706f4d4afec-kube-api-access-2tlfm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:55.933360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.933073 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6acab08b-2437-4f42-8c45-f706f4d4afec-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:55.982170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.982138 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:30:55.982611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.982543 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:30:55.983255 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:55.983239 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:30:56.546991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:56.546961 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" Nov 02 20:30:56.546991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:56.546983 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm" event={"ID":"6acab08b-2437-4f42-8c45-f706f4d4afec","Type":"ContainerDied","Data":"af3f5ece5f53e5c1f855d01e730d5a971d68199bf432ccffa90a7abeef17509e"} Nov 02 20:30:56.547267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:30:56.547042 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af3f5ece5f53e5c1f855d01e730d5a971d68199bf432ccffa90a7abeef17509e" Nov 02 20:31:08.258181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.258143 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc"] Nov 02 20:31:08.258727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.258712 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6acab08b-2437-4f42-8c45-f706f4d4afec" containerName="revoke-keys" Nov 02 20:31:08.258774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.258728 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6acab08b-2437-4f42-8c45-f706f4d4afec" containerName="revoke-keys" Nov 02 20:31:08.258833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.258824 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6acab08b-2437-4f42-8c45-f706f4d4afec" containerName="revoke-keys" Nov 02 20:31:08.263409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.263390 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.314583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.314547 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc"] Nov 02 20:31:08.356966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.356938 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:31:08.357169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.357064 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:08.357169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.357080 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:31:08.358570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.358557 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:31:08.395926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.395884 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjsxt\" (UniqueName: \"kubernetes.io/projected/4094d024-75b2-4c15-9f6d-e21e17fa36fe-kube-api-access-hjsxt\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.396098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.396004 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.396098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.396060 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.497739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.497701 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.497934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.497765 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.497934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.497877 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjsxt\" (UniqueName: \"kubernetes.io/projected/4094d024-75b2-4c15-9f6d-e21e17fa36fe-kube-api-access-hjsxt\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.498578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.498547 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.500654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.500631 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.551426 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.551353 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjsxt\" (UniqueName: \"kubernetes.io/projected/4094d024-75b2-4c15-9f6d-e21e17fa36fe-kube-api-access-hjsxt\") pod \"maas-api-revoke-sub-94c63bd5-twhrc\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.577332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.577272 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:08.948883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:08.948843 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc"] Nov 02 20:31:08.949980 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:31:08.949952 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4094d024_75b2_4c15_9f6d_e21e17fa36fe.slice/crio-19a2caf528ab3aec4d7de751432375159649b1f86ff2fc6a59c9ee60a0f76488 WatchSource:0}: Error finding container 19a2caf528ab3aec4d7de751432375159649b1f86ff2fc6a59c9ee60a0f76488: Status 404 returned error can't find the container with id 19a2caf528ab3aec4d7de751432375159649b1f86ff2fc6a59c9ee60a0f76488 Nov 02 20:31:09.648359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:09.648316 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" event={"ID":"4094d024-75b2-4c15-9f6d-e21e17fa36fe","Type":"ContainerStarted","Data":"d5ed1ea299b181dd8746794ab7a486dee013155bb20cd080b2df2ea11e2eb8db"} Nov 02 20:31:09.648840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:09.648375 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" event={"ID":"4094d024-75b2-4c15-9f6d-e21e17fa36fe","Type":"ContainerStarted","Data":"19a2caf528ab3aec4d7de751432375159649b1f86ff2fc6a59c9ee60a0f76488"} Nov 02 20:31:09.755736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:09.755687 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" podStartSLOduration=1.7556748039999999 podStartE2EDuration="1.755674804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:09.753309879 +0000 UTC m=+2568.270082460" watchObservedRunningTime="2026-11-02 20:31:09.755674804 +0000 UTC m=+2568.272447389" Nov 02 20:31:10.658612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:10.658572 5800 generic.go:356] "Generic (PLEG): container finished" podID="4094d024-75b2-4c15-9f6d-e21e17fa36fe" containerID="d5ed1ea299b181dd8746794ab7a486dee013155bb20cd080b2df2ea11e2eb8db" exitCode=0 Nov 02 20:31:10.659039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:10.658652 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" event={"ID":"4094d024-75b2-4c15-9f6d-e21e17fa36fe","Type":"ContainerDied","Data":"d5ed1ea299b181dd8746794ab7a486dee013155bb20cd080b2df2ea11e2eb8db"} Nov 02 20:31:11.825611 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:11.825580 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:11.938828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:11.938749 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-xks-ca\") pod \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " Nov 02 20:31:11.938828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:11.938821 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-openshift-ca\") pod \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " Nov 02 20:31:11.939048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:11.938992 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4094d024-75b2-4c15-9f6d-e21e17fa36fe-kube-api-access-hjsxt\" (UniqueName: \"kubernetes.io/projected/4094d024-75b2-4c15-9f6d-e21e17fa36fe-kube-api-access-hjsxt\") pod \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\" (UID: \"4094d024-75b2-4c15-9f6d-e21e17fa36fe\") " Nov 02 20:31:11.939517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:11.939472 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-openshift-ca" pod "4094d024-75b2-4c15-9f6d-e21e17fa36fe" (UID: "4094d024-75b2-4c15-9f6d-e21e17fa36fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:11.941587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:11.941560 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4094d024-75b2-4c15-9f6d-e21e17fa36fe-kube-api-access-hjsxt" pod "4094d024-75b2-4c15-9f6d-e21e17fa36fe" (UID: "4094d024-75b2-4c15-9f6d-e21e17fa36fe"). InnerVolumeSpecName "kube-api-access-hjsxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:11.941694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:11.941646 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-xks-ca" pod "4094d024-75b2-4c15-9f6d-e21e17fa36fe" (UID: "4094d024-75b2-4c15-9f6d-e21e17fa36fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:12.041137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:12.041097 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjsxt\" (UniqueName: \"kubernetes.io/projected/4094d024-75b2-4c15-9f6d-e21e17fa36fe-kube-api-access-hjsxt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:12.041137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:12.041138 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:12.041373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:12.041157 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4094d024-75b2-4c15-9f6d-e21e17fa36fe-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:12.677793 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:12.677766 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" Nov 02 20:31:12.677976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:12.677758 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc" event={"ID":"4094d024-75b2-4c15-9f6d-e21e17fa36fe","Type":"ContainerDied","Data":"19a2caf528ab3aec4d7de751432375159649b1f86ff2fc6a59c9ee60a0f76488"} Nov 02 20:31:12.677976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:12.677891 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19a2caf528ab3aec4d7de751432375159649b1f86ff2fc6a59c9ee60a0f76488" Nov 02 20:31:22.186815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:22.186778 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27"] Nov 02 20:31:22.289049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:22.289010 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9"] Nov 02 20:31:22.714124 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:22.714061 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:31:24.412958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.412921 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf"] Nov 02 20:31:24.413712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.413688 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4094d024-75b2-4c15-9f6d-e21e17fa36fe" containerName="revoke-keys" Nov 02 20:31:24.413828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.413713 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4094d024-75b2-4c15-9f6d-e21e17fa36fe" containerName="revoke-keys" Nov 02 20:31:24.413890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.413882 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4094d024-75b2-4c15-9f6d-e21e17fa36fe" containerName="revoke-keys" Nov 02 20:31:24.418838 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.418816 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.470852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.470818 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf"] Nov 02 20:31:24.486068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.486030 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.486350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.486163 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.486350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.486255 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkn99\" (UniqueName: \"kubernetes.io/projected/4b57839f-6f4c-4595-865d-84c1a1f42ba4-kube-api-access-lkn99\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.514373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.514341 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:31:24.514373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.514340 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:31:24.560896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.560856 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:24.561082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.560934 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:31:24.588037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.588001 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.588227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.588139 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.588317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.588243 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkn99\" (UniqueName: \"kubernetes.io/projected/4b57839f-6f4c-4595-865d-84c1a1f42ba4-kube-api-access-lkn99\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.588927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.588899 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.590674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.590649 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.642185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.642149 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkn99\" (UniqueName: \"kubernetes.io/projected/4b57839f-6f4c-4595-865d-84c1a1f42ba4-kube-api-access-lkn99\") pod \"maas-api-revoke-sub-07aeb0f1-m2ldf\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:24.733136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:24.733050 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:25.106796 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:25.106769 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf"] Nov 02 20:31:25.107637 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:31:25.107612 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b57839f_6f4c_4595_865d_84c1a1f42ba4.slice/crio-ecd3e5db9466ecb0b048e82160487cd2bb4fa06c8305e4490a28806f857a4594 WatchSource:0}: Error finding container ecd3e5db9466ecb0b048e82160487cd2bb4fa06c8305e4490a28806f857a4594: Status 404 returned error can't find the container with id ecd3e5db9466ecb0b048e82160487cd2bb4fa06c8305e4490a28806f857a4594 Nov 02 20:31:25.773186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:25.773145 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" event={"ID":"4b57839f-6f4c-4595-865d-84c1a1f42ba4","Type":"ContainerStarted","Data":"95c59f7dba9ffa9e3a9fb88855a0b171a67698ffa42ef3d58ebb81988f664e52"} Nov 02 20:31:25.773741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:25.773214 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" event={"ID":"4b57839f-6f4c-4595-865d-84c1a1f42ba4","Type":"ContainerStarted","Data":"ecd3e5db9466ecb0b048e82160487cd2bb4fa06c8305e4490a28806f857a4594"} Nov 02 20:31:25.881138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:25.881077 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" podStartSLOduration=1.881063218 podStartE2EDuration="1.881063218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:25.878442778 +0000 UTC m=+2584.395215363" watchObservedRunningTime="2026-11-02 20:31:25.881063218 +0000 UTC m=+2584.397835794" Nov 02 20:31:26.717512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:26.717476 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml"] Nov 02 20:31:26.717708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:26.717669 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" containerID="cri-o://8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c" gracePeriod=30 Nov 02 20:31:26.782152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:26.782110 5800 generic.go:356] "Generic (PLEG): container finished" podID="4b57839f-6f4c-4595-865d-84c1a1f42ba4" containerID="95c59f7dba9ffa9e3a9fb88855a0b171a67698ffa42ef3d58ebb81988f664e52" exitCode=0 Nov 02 20:31:26.782590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:26.782197 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" event={"ID":"4b57839f-6f4c-4595-865d-84c1a1f42ba4","Type":"ContainerDied","Data":"95c59f7dba9ffa9e3a9fb88855a0b171a67698ffa42ef3d58ebb81988f664e52"} Nov 02 20:31:27.187253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.187205 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" podUID="cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" containerName="payload-processing" containerID="cri-o://1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2" gracePeriod=25 Nov 02 20:31:27.290004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.289951 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" podUID="1a5220a8-dfa9-4597-9d6c-4b307bf78a64" containerName="payload-pre-processing" containerID="cri-o://1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504" gracePeriod=25 Nov 02 20:31:27.505886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.505852 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:31:27.524782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.524755 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-plugins-config-volume\") pod \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " Nov 02 20:31:27.525047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.525031 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-kube-api-access-pm97n\" (UniqueName: \"kubernetes.io/projected/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-kube-api-access-pm97n\") pod \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\" (UID: \"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff\") " Nov 02 20:31:27.525140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.525110 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-plugins-config-volume" pod "cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" (UID: "cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:27.525433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.525409 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:27.527537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.527508 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-kube-api-access-pm97n" pod "cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" (UID: "cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff"). InnerVolumeSpecName "kube-api-access-pm97n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:27.601545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.601515 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:31:27.626551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.626523 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-kube-api-access-sw728\" (UniqueName: \"kubernetes.io/projected/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-kube-api-access-sw728\") pod \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " Nov 02 20:31:27.626716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.626600 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-plugins-config-volume\") pod \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\" (UID: \"1a5220a8-dfa9-4597-9d6c-4b307bf78a64\") " Nov 02 20:31:27.627492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.627031 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm97n\" (UniqueName: \"kubernetes.io/projected/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff-kube-api-access-pm97n\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:27.627492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.627257 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-plugins-config-volume" pod "1a5220a8-dfa9-4597-9d6c-4b307bf78a64" (UID: "1a5220a8-dfa9-4597-9d6c-4b307bf78a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:27.630130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.630099 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-kube-api-access-sw728" pod "1a5220a8-dfa9-4597-9d6c-4b307bf78a64" (UID: "1a5220a8-dfa9-4597-9d6c-4b307bf78a64"). InnerVolumeSpecName "kube-api-access-sw728". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:27.728347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.727975 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw728\" (UniqueName: \"kubernetes.io/projected/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-kube-api-access-sw728\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:27.728347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.728011 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5220a8-dfa9-4597-9d6c-4b307bf78a64-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:27.790725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.790691 5800 generic.go:356] "Generic (PLEG): container finished" podID="cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" containerID="1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2" exitCode=0 Nov 02 20:31:27.791192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.790741 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" Nov 02 20:31:27.791192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.790781 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" event={"ID":"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff","Type":"ContainerDied","Data":"1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2"} Nov 02 20:31:27.791192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.790835 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27" event={"ID":"cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff","Type":"ContainerDied","Data":"a8d0a43a8e959b4bc4f027372757a863bdc590dce449b21270e8868d887c9d87"} Nov 02 20:31:27.791192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.790866 5800 scope.go:122] "RemoveContainer" containerID="1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2" Nov 02 20:31:27.792217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.792197 5800 generic.go:356] "Generic (PLEG): container finished" podID="1a5220a8-dfa9-4597-9d6c-4b307bf78a64" containerID="1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504" exitCode=0 Nov 02 20:31:27.792275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.792258 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" Nov 02 20:31:27.792362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.792277 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" event={"ID":"1a5220a8-dfa9-4597-9d6c-4b307bf78a64","Type":"ContainerDied","Data":"1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504"} Nov 02 20:31:27.792362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.792347 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9" event={"ID":"1a5220a8-dfa9-4597-9d6c-4b307bf78a64","Type":"ContainerDied","Data":"8ab34e310f2f3ce8e5904c7aee7dbecb8a289756ba9b98a8fc6a5c353b83db28"} Nov 02 20:31:27.803423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.803403 5800 scope.go:122] "RemoveContainer" containerID="1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2" Nov 02 20:31:27.803730 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:27.803711 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2\": container with ID starting with 1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2 not found: ID does not exist" containerID="1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2" Nov 02 20:31:27.803806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.803737 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2"} err="failed to get container status \"1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2\": rpc error: code = NotFound desc = could not find container \"1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2\": container with ID starting with 1ce88b4937ecc8617e73c0516bdc5d35911558ee849ff0db9e2d1865f48618b2 not found: ID does not exist" Nov 02 20:31:27.803806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.803756 5800 scope.go:122] "RemoveContainer" containerID="1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504" Nov 02 20:31:27.813455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.813432 5800 scope.go:122] "RemoveContainer" containerID="1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504" Nov 02 20:31:27.813752 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:27.813731 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504\": container with ID starting with 1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504 not found: ID does not exist" containerID="1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504" Nov 02 20:31:27.813832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.813760 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504"} err="failed to get container status \"1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504\": rpc error: code = NotFound desc = could not find container \"1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504\": container with ID starting with 1fa1c01f85160c76398f6531ecbeab7d55bb19d88bee113011b660a9d3255504 not found: ID does not exist" Nov 02 20:31:27.942367 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.942336 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:27.946023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.945998 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27"] Nov 02 20:31:27.949200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:27.949177 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7cad76-69dc6c8d4c-hzt27"] Nov 02 20:31:28.031168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.031135 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b57839f-6f4c-4595-865d-84c1a1f42ba4-kube-api-access-lkn99\" (UniqueName: \"kubernetes.io/projected/4b57839f-6f4c-4595-865d-84c1a1f42ba4-kube-api-access-lkn99\") pod \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " Nov 02 20:31:28.031364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.031258 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-openshift-ca\") pod \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " Nov 02 20:31:28.031438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.031421 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-xks-ca\") pod \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\" (UID: \"4b57839f-6f4c-4595-865d-84c1a1f42ba4\") " Nov 02 20:31:28.031642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.031613 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-openshift-ca" pod "4b57839f-6f4c-4595-865d-84c1a1f42ba4" (UID: "4b57839f-6f4c-4595-865d-84c1a1f42ba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:28.031887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.031871 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.033844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.033822 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b57839f-6f4c-4595-865d-84c1a1f42ba4-kube-api-access-lkn99" pod "4b57839f-6f4c-4595-865d-84c1a1f42ba4" (UID: "4b57839f-6f4c-4595-865d-84c1a1f42ba4"). InnerVolumeSpecName "kube-api-access-lkn99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:28.033844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.033830 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-xks-ca" pod "4b57839f-6f4c-4595-865d-84c1a1f42ba4" (UID: "4b57839f-6f4c-4595-865d-84c1a1f42ba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:28.049563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.049536 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9"] Nov 02 20:31:28.054291 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.054255 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7cad76-597b9c84bc-flhz9"] Nov 02 20:31:28.059425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.059400 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:31:28.059514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.059459 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:31:28.132561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.132523 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b57839f-6f4c-4595-865d-84c1a1f42ba4-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.132561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.132559 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkn99\" (UniqueName: \"kubernetes.io/projected/4b57839f-6f4c-4595-865d-84c1a1f42ba4-kube-api-access-lkn99\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:28.800425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.800396 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" Nov 02 20:31:28.800425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.800405 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf" event={"ID":"4b57839f-6f4c-4595-865d-84c1a1f42ba4","Type":"ContainerDied","Data":"ecd3e5db9466ecb0b048e82160487cd2bb4fa06c8305e4490a28806f857a4594"} Nov 02 20:31:28.800926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:28.800454 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecd3e5db9466ecb0b048e82160487cd2bb4fa06c8305e4490a28806f857a4594" Nov 02 20:31:29.624941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:29.624903 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5220a8-dfa9-4597-9d6c-4b307bf78a64" path="/var/lib/kubelet/pods/1a5220a8-dfa9-4597-9d6c-4b307bf78a64/volumes" Nov 02 20:31:29.625464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:29.625442 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" path="/var/lib/kubelet/pods/cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff/volumes" Nov 02 20:31:31.999704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:31.999679 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:31:32.081385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081346 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-kube-api-access-l96c9\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-kube-api-access-l96c9\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081404 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-data\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081435 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istiod-ca-cert\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081514 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-credential-socket\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081569 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-socket\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081606 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-podinfo\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081644 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-token\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-token\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081695 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-envoy\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.081805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081733 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-data" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:32.082022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081838 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-credential-socket" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:32.082022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081850 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-socket" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:32.082022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081898 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-certs\") pod \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\" (UID: \"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15\") " Nov 02 20:31:32.082022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.081946 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istiod-ca-cert" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:32.082265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.082233 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-certs" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:32.082529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.082508 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.082620 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.082536 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.082620 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.082553 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.082620 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.082568 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.082620 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.082583 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.083870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.083842 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-envoy" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:32.083998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.083874 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-podinfo" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:32.083998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.083899 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-kube-api-access-l96c9" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "kube-api-access-l96c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:32.084161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.084144 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-token" pod "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" (UID: "4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:32.190327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.185619 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.190327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.185657 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.190327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.185672 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.190327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.185686 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96c9\" (UniqueName: \"kubernetes.io/projected/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15-kube-api-access-l96c9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:32.830108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.830065 5800 generic.go:356] "Generic (PLEG): container finished" podID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerID="8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c" exitCode=0 Nov 02 20:31:32.830329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.830119 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" Nov 02 20:31:32.830329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.830134 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" event={"ID":"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15","Type":"ContainerDied","Data":"8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c"} Nov 02 20:31:32.830329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.830183 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml" event={"ID":"4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15","Type":"ContainerDied","Data":"cf8ec39f0848dccc72d598a5349e938664a618b45d3e5b085a524b04062b0c8d"} Nov 02 20:31:32.830329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.830205 5800 scope.go:122] "RemoveContainer" containerID="8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c" Nov 02 20:31:32.841547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.841351 5800 scope.go:122] "RemoveContainer" containerID="4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2" Nov 02 20:31:32.850953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.850936 5800 scope.go:122] "RemoveContainer" containerID="8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c" Nov 02 20:31:32.851229 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:32.851208 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c\": container with ID starting with 8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c not found: ID does not exist" containerID="8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c" Nov 02 20:31:32.851317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.851236 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c"} err="failed to get container status \"8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c\": rpc error: code = NotFound desc = could not find container \"8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c\": container with ID starting with 8479c80b485911e51134dc87fef158c3505fb0e9e4a5480486812b8aea6d782c not found: ID does not exist" Nov 02 20:31:32.851317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.851253 5800 scope.go:122] "RemoveContainer" containerID="4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2" Nov 02 20:31:32.851583 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:32.851565 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2\": container with ID starting with 4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2 not found: ID does not exist" containerID="4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2" Nov 02 20:31:32.851640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.851589 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2"} err="failed to get container status \"4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2\": rpc error: code = NotFound desc = could not find container \"4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2\": container with ID starting with 4855929c0173343c30a3ebedd0b647bc9235bc7fa9988d0f414e198b61cee7f2 not found: ID does not exist" Nov 02 20:31:32.984990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:32.984959 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml"] Nov 02 20:31:33.030964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:33.030926 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdbffd-openshift-default-5b747c7b57-6lpml"] Nov 02 20:31:33.621678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:33.621645 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" path="/var/lib/kubelet/pods/4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15/volumes" Nov 02 20:31:38.759113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759074 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl"] Nov 02 20:31:38.759697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759676 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4b57839f-6f4c-4595-865d-84c1a1f42ba4" containerName="revoke-keys" Nov 02 20:31:38.759697 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759697 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b57839f-6f4c-4595-865d-84c1a1f42ba4" containerName="revoke-keys" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759711 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" containerName="payload-processing" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759717 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" containerName="payload-processing" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759728 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1a5220a8-dfa9-4597-9d6c-4b307bf78a64" containerName="payload-pre-processing" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759734 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5220a8-dfa9-4597-9d6c-4b307bf78a64" containerName="payload-pre-processing" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759757 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759762 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759780 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" Nov 02 20:31:38.759827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759786 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" Nov 02 20:31:38.760244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759862 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" Nov 02 20:31:38.760244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759872 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ceafdd2-5bea-4a23-a8a5-89c34ddeaa15" containerName="istio-proxy" Nov 02 20:31:38.760244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759879 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b57839f-6f4c-4595-865d-84c1a1f42ba4" containerName="revoke-keys" Nov 02 20:31:38.760244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759886 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3c76e3-a6d1-40d8-80a3-b4dda1ea0eff" containerName="payload-processing" Nov 02 20:31:38.760244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.759893 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5220a8-dfa9-4597-9d6c-4b307bf78a64" containerName="payload-pre-processing" Nov 02 20:31:38.764732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.764706 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.816320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.816270 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl"] Nov 02 20:31:38.858615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.858571 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:31:38.858615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.858587 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:38.859049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.859024 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:31:38.859901 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.859848 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:31:38.860453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.860425 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgch2\" (UniqueName: \"kubernetes.io/projected/52dd166b-634f-456d-8488-89333dedb698-kube-api-access-vgch2\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.860636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.860562 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dd166b-634f-456d-8488-89333dedb698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.860772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.860752 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dd166b-634f-456d-8488-89333dedb698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.972937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.969583 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dd166b-634f-456d-8488-89333dedb698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.972937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.969688 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dd166b-634f-456d-8488-89333dedb698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.972937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.969794 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgch2\" (UniqueName: \"kubernetes.io/projected/52dd166b-634f-456d-8488-89333dedb698-kube-api-access-vgch2\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.972937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.971442 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dd166b-634f-456d-8488-89333dedb698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:38.975250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:38.974269 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dd166b-634f-456d-8488-89333dedb698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:39.024002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:39.023923 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgch2\" (UniqueName: \"kubernetes.io/projected/52dd166b-634f-456d-8488-89333dedb698-kube-api-access-vgch2\") pod \"maas-api-revoke-sub-591fa8e0-tx2bl\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:39.079752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:39.079713 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:39.449540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:39.449508 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl"] Nov 02 20:31:39.450481 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:31:39.450451 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dd166b_634f_456d_8488_89333dedb698.slice/crio-091c6701cbdc573306a139d3d6462784891bac2899145f3f6eff0797343d1a99 WatchSource:0}: Error finding container 091c6701cbdc573306a139d3d6462784891bac2899145f3f6eff0797343d1a99: Status 404 returned error can't find the container with id 091c6701cbdc573306a139d3d6462784891bac2899145f3f6eff0797343d1a99 Nov 02 20:31:39.880250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:39.880211 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" event={"ID":"52dd166b-634f-456d-8488-89333dedb698","Type":"ContainerStarted","Data":"a5ca74226ef4c33d94c87f251220e8f7ffa65151f7f07e478d0be3ac5d12a992"} Nov 02 20:31:39.880636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:39.880275 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" event={"ID":"52dd166b-634f-456d-8488-89333dedb698","Type":"ContainerStarted","Data":"091c6701cbdc573306a139d3d6462784891bac2899145f3f6eff0797343d1a99"} Nov 02 20:31:39.987374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:39.987257 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" podStartSLOduration=1.98724289 podStartE2EDuration="1.98724289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:39.985621365 +0000 UTC m=+2598.502393952" watchObservedRunningTime="2026-11-02 20:31:39.98724289 +0000 UTC m=+2598.504015475" Nov 02 20:31:40.643106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.643066 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8"] Nov 02 20:31:40.647083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.647066 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.699162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.699084 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8"] Nov 02 20:31:40.790399 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.790360 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.790399 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.790400 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.790633 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.790613 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqnnd\" (UniqueName: \"kubernetes.io/projected/5b3722ae-b555-4a51-9784-0bb19aaa8c24-kube-api-access-sqnnd\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.888100 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.888065 5800 generic.go:356] "Generic (PLEG): container finished" podID="52dd166b-634f-456d-8488-89333dedb698" containerID="a5ca74226ef4c33d94c87f251220e8f7ffa65151f7f07e478d0be3ac5d12a992" exitCode=0 Nov 02 20:31:40.888527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.888152 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" event={"ID":"52dd166b-634f-456d-8488-89333dedb698","Type":"ContainerDied","Data":"a5ca74226ef4c33d94c87f251220e8f7ffa65151f7f07e478d0be3ac5d12a992"} Nov 02 20:31:40.891707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.891681 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqnnd\" (UniqueName: \"kubernetes.io/projected/5b3722ae-b555-4a51-9784-0bb19aaa8c24-kube-api-access-sqnnd\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.891844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.891782 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.891844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.891813 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.892522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.892500 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.894075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.894047 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.944710 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.944675 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqnnd\" (UniqueName: \"kubernetes.io/projected/5b3722ae-b555-4a51-9784-0bb19aaa8c24-kube-api-access-sqnnd\") pod \"maas-api-revoke-sub-2b9a6631-tc6r8\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:40.961391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:40.961318 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:41.332965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:41.332933 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8"] Nov 02 20:31:41.334012 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:31:41.333980 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3722ae_b555_4a51_9784_0bb19aaa8c24.slice/crio-e0f18240033b05bc5d1c034b755e930772e84bb99a13de5eead1f3b35123d3ee WatchSource:0}: Error finding container e0f18240033b05bc5d1c034b755e930772e84bb99a13de5eead1f3b35123d3ee: Status 404 returned error can't find the container with id e0f18240033b05bc5d1c034b755e930772e84bb99a13de5eead1f3b35123d3ee Nov 02 20:31:41.896363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:41.896240 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" event={"ID":"5b3722ae-b555-4a51-9784-0bb19aaa8c24","Type":"ContainerStarted","Data":"fb7d9c197be13dec897fb21fb57460361738dacd56e9d6002618988b6b7fb7b0"} Nov 02 20:31:41.896363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:41.896320 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" event={"ID":"5b3722ae-b555-4a51-9784-0bb19aaa8c24","Type":"ContainerStarted","Data":"e0f18240033b05bc5d1c034b755e930772e84bb99a13de5eead1f3b35123d3ee"} Nov 02 20:31:42.043676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.043650 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:42.154270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.154161 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" podStartSLOduration=2.154144158 podStartE2EDuration="2.154144158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:42.00449485 +0000 UTC m=+2600.521267435" watchObservedRunningTime="2026-11-02 20:31:42.154144158 +0000 UTC m=+2600.670916746" Nov 02 20:31:42.207346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.207312 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52dd166b-634f-456d-8488-89333dedb698-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dd166b-634f-456d-8488-89333dedb698-maas-api-openshift-ca\") pod \"52dd166b-634f-456d-8488-89333dedb698\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " Nov 02 20:31:42.207509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.207409 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52dd166b-634f-456d-8488-89333dedb698-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dd166b-634f-456d-8488-89333dedb698-maas-api-xks-ca\") pod \"52dd166b-634f-456d-8488-89333dedb698\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " Nov 02 20:31:42.207629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.207609 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52dd166b-634f-456d-8488-89333dedb698-kube-api-access-vgch2\" (UniqueName: \"kubernetes.io/projected/52dd166b-634f-456d-8488-89333dedb698-kube-api-access-vgch2\") pod \"52dd166b-634f-456d-8488-89333dedb698\" (UID: \"52dd166b-634f-456d-8488-89333dedb698\") " Nov 02 20:31:42.207693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.207669 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52dd166b-634f-456d-8488-89333dedb698-maas-api-openshift-ca" pod "52dd166b-634f-456d-8488-89333dedb698" (UID: "52dd166b-634f-456d-8488-89333dedb698"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:42.207987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.207966 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52dd166b-634f-456d-8488-89333dedb698-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:42.209623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.209601 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52dd166b-634f-456d-8488-89333dedb698-kube-api-access-vgch2" pod "52dd166b-634f-456d-8488-89333dedb698" (UID: "52dd166b-634f-456d-8488-89333dedb698"). InnerVolumeSpecName "kube-api-access-vgch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:42.209700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.209631 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52dd166b-634f-456d-8488-89333dedb698-maas-api-xks-ca" pod "52dd166b-634f-456d-8488-89333dedb698" (UID: "52dd166b-634f-456d-8488-89333dedb698"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:42.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.308928 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgch2\" (UniqueName: \"kubernetes.io/projected/52dd166b-634f-456d-8488-89333dedb698-kube-api-access-vgch2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:42.308963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.308959 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52dd166b-634f-456d-8488-89333dedb698-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:42.904830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.904800 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" Nov 02 20:31:42.905315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.904840 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl" event={"ID":"52dd166b-634f-456d-8488-89333dedb698","Type":"ContainerDied","Data":"091c6701cbdc573306a139d3d6462784891bac2899145f3f6eff0797343d1a99"} Nov 02 20:31:42.905315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.904892 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091c6701cbdc573306a139d3d6462784891bac2899145f3f6eff0797343d1a99" Nov 02 20:31:42.906688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.906661 5800 generic.go:356] "Generic (PLEG): container finished" podID="5b3722ae-b555-4a51-9784-0bb19aaa8c24" containerID="fb7d9c197be13dec897fb21fb57460361738dacd56e9d6002618988b6b7fb7b0" exitCode=0 Nov 02 20:31:42.906801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:42.906765 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" event={"ID":"5b3722ae-b555-4a51-9784-0bb19aaa8c24","Type":"ContainerDied","Data":"fb7d9c197be13dec897fb21fb57460361738dacd56e9d6002618988b6b7fb7b0"} Nov 02 20:31:44.121534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.121498 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:44.241802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.241712 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-openshift-ca\") pod \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " Nov 02 20:31:44.241996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.241806 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3722ae-b555-4a51-9784-0bb19aaa8c24-kube-api-access-sqnnd\" (UniqueName: \"kubernetes.io/projected/5b3722ae-b555-4a51-9784-0bb19aaa8c24-kube-api-access-sqnnd\") pod \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " Nov 02 20:31:44.241996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.241903 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-xks-ca\") pod \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\" (UID: \"5b3722ae-b555-4a51-9784-0bb19aaa8c24\") " Nov 02 20:31:44.242155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.242132 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-openshift-ca" pod "5b3722ae-b555-4a51-9784-0bb19aaa8c24" (UID: "5b3722ae-b555-4a51-9784-0bb19aaa8c24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:44.242466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.242441 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.244082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.244054 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3722ae-b555-4a51-9784-0bb19aaa8c24-kube-api-access-sqnnd" pod "5b3722ae-b555-4a51-9784-0bb19aaa8c24" (UID: "5b3722ae-b555-4a51-9784-0bb19aaa8c24"). InnerVolumeSpecName "kube-api-access-sqnnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:44.244190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.244120 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-xks-ca" pod "5b3722ae-b555-4a51-9784-0bb19aaa8c24" (UID: "5b3722ae-b555-4a51-9784-0bb19aaa8c24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:44.343920 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.343871 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqnnd\" (UniqueName: \"kubernetes.io/projected/5b3722ae-b555-4a51-9784-0bb19aaa8c24-kube-api-access-sqnnd\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.343920 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.343914 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3722ae-b555-4a51-9784-0bb19aaa8c24-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.923429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.923400 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" Nov 02 20:31:44.923626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.923427 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8" event={"ID":"5b3722ae-b555-4a51-9784-0bb19aaa8c24","Type":"ContainerDied","Data":"e0f18240033b05bc5d1c034b755e930772e84bb99a13de5eead1f3b35123d3ee"} Nov 02 20:31:44.923626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:44.923488 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f18240033b05bc5d1c034b755e930772e84bb99a13de5eead1f3b35123d3ee" Nov 02 20:31:51.537136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.537096 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp"] Nov 02 20:31:51.537903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.537881 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5b3722ae-b555-4a51-9784-0bb19aaa8c24" containerName="revoke-keys" Nov 02 20:31:51.537952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.537906 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3722ae-b555-4a51-9784-0bb19aaa8c24" containerName="revoke-keys" Nov 02 20:31:51.537952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.537921 5800 cpu_manager.go:414] "containerMap: removing container" podUID="52dd166b-634f-456d-8488-89333dedb698" containerName="revoke-keys" Nov 02 20:31:51.537952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.537930 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52dd166b-634f-456d-8488-89333dedb698" containerName="revoke-keys" Nov 02 20:31:51.538119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.538107 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="52dd166b-634f-456d-8488-89333dedb698" containerName="revoke-keys" Nov 02 20:31:51.538155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.538136 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3722ae-b555-4a51-9784-0bb19aaa8c24" containerName="revoke-keys" Nov 02 20:31:51.542207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.542187 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.592473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.592430 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp"] Nov 02 20:31:51.626135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.626092 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.626328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.626238 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtkx\" (UniqueName: \"kubernetes.io/projected/3986517e-1d56-42d0-9e97-35f6e18eab05-kube-api-access-hmtkx\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.626432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.626405 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.635267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.635239 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:31:51.635419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.635239 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:31:51.635419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.635244 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:31:51.635419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.635365 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:31:51.730014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.729979 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.730233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.730120 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.730233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.730208 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtkx\" (UniqueName: \"kubernetes.io/projected/3986517e-1d56-42d0-9e97-35f6e18eab05-kube-api-access-hmtkx\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.731345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.731274 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.732972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.732940 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.783777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.783736 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtkx\" (UniqueName: \"kubernetes.io/projected/3986517e-1d56-42d0-9e97-35f6e18eab05-kube-api-access-hmtkx\") pod \"maas-api-revoke-sub-54cd9d82-lzmtp\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:51.856309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:51.856209 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:52.239424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:52.239392 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp"] Nov 02 20:31:52.240505 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:31:52.240479 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3986517e_1d56_42d0_9e97_35f6e18eab05.slice/crio-3d1ecf90f803049623ec2288e264090c8ba973db3cddaf78be841beefb70812d WatchSource:0}: Error finding container 3d1ecf90f803049623ec2288e264090c8ba973db3cddaf78be841beefb70812d: Status 404 returned error can't find the container with id 3d1ecf90f803049623ec2288e264090c8ba973db3cddaf78be841beefb70812d Nov 02 20:31:52.337794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:52.337753 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb"] Nov 02 20:31:52.340914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:52.338409 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" containerID="cri-o://8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6" gracePeriod=30 Nov 02 20:31:52.697841 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:52.697805 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:31:52.989887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:52.989787 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" event={"ID":"3986517e-1d56-42d0-9e97-35f6e18eab05","Type":"ContainerStarted","Data":"0ea39df8a690180644c7f872a20412d1f8dafa4fdb9c3171ee0b9ea8f23e25a7"} Nov 02 20:31:52.989887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:52.989883 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" event={"ID":"3986517e-1d56-42d0-9e97-35f6e18eab05","Type":"ContainerStarted","Data":"3d1ecf90f803049623ec2288e264090c8ba973db3cddaf78be841beefb70812d"} Nov 02 20:31:53.098390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:53.098338 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" podStartSLOduration=2.098320744 podStartE2EDuration="2.098320744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:53.095969116 +0000 UTC m=+2611.612741701" watchObservedRunningTime="2026-11-02 20:31:53.098320744 +0000 UTC m=+2611.615093330" Nov 02 20:31:54.004118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.004077 5800 generic.go:356] "Generic (PLEG): container finished" podID="3986517e-1d56-42d0-9e97-35f6e18eab05" containerID="0ea39df8a690180644c7f872a20412d1f8dafa4fdb9c3171ee0b9ea8f23e25a7" exitCode=0 Nov 02 20:31:54.004678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.004154 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" event={"ID":"3986517e-1d56-42d0-9e97-35f6e18eab05","Type":"ContainerDied","Data":"0ea39df8a690180644c7f872a20412d1f8dafa4fdb9c3171ee0b9ea8f23e25a7"} Nov 02 20:31:54.593327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.593263 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56"] Nov 02 20:31:54.597993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.597968 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.649976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.649942 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56"] Nov 02 20:31:54.672241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.672209 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.672453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.672251 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzl95\" (UniqueName: \"kubernetes.io/projected/36dfbe42-2a12-47c3-80d6-5616a232b401-kube-api-access-rzl95\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.672453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.672417 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.774270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.774226 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.774270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.774271 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzl95\" (UniqueName: \"kubernetes.io/projected/36dfbe42-2a12-47c3-80d6-5616a232b401-kube-api-access-rzl95\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.774544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.774358 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.775092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.775069 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.776704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.776685 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.827449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.827412 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzl95\" (UniqueName: \"kubernetes.io/projected/36dfbe42-2a12-47c3-80d6-5616a232b401-kube-api-access-rzl95\") pod \"maas-api-revoke-sub-ac5bdd5e-z8h56\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:54.912176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:54.912095 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:55.222843 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.222813 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:55.280505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.280476 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56"] Nov 02 20:31:55.282798 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:31:55.282767 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dfbe42_2a12_47c3_80d6_5616a232b401.slice/crio-d979f9608decee38ebbd29c82c7e131cf0910e02a4a142e0284f20a319c2ca71 WatchSource:0}: Error finding container d979f9608decee38ebbd29c82c7e131cf0910e02a4a142e0284f20a319c2ca71: Status 404 returned error can't find the container with id d979f9608decee38ebbd29c82c7e131cf0910e02a4a142e0284f20a319c2ca71 Nov 02 20:31:55.291228 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.291203 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-xks-ca\") pod \"3986517e-1d56-42d0-9e97-35f6e18eab05\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " Nov 02 20:31:55.291384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.291369 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-openshift-ca\") pod \"3986517e-1d56-42d0-9e97-35f6e18eab05\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " Nov 02 20:31:55.291458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.291446 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3986517e-1d56-42d0-9e97-35f6e18eab05-kube-api-access-hmtkx\" (UniqueName: \"kubernetes.io/projected/3986517e-1d56-42d0-9e97-35f6e18eab05-kube-api-access-hmtkx\") pod \"3986517e-1d56-42d0-9e97-35f6e18eab05\" (UID: \"3986517e-1d56-42d0-9e97-35f6e18eab05\") " Nov 02 20:31:55.291727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.291702 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-openshift-ca" pod "3986517e-1d56-42d0-9e97-35f6e18eab05" (UID: "3986517e-1d56-42d0-9e97-35f6e18eab05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:55.291802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.291794 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:55.293500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.293478 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3986517e-1d56-42d0-9e97-35f6e18eab05-kube-api-access-hmtkx" pod "3986517e-1d56-42d0-9e97-35f6e18eab05" (UID: "3986517e-1d56-42d0-9e97-35f6e18eab05"). InnerVolumeSpecName "kube-api-access-hmtkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:55.293748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.293728 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-xks-ca" pod "3986517e-1d56-42d0-9e97-35f6e18eab05" (UID: "3986517e-1d56-42d0-9e97-35f6e18eab05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:55.393138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.393101 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3986517e-1d56-42d0-9e97-35f6e18eab05-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:55.393138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:55.393133 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmtkx\" (UniqueName: \"kubernetes.io/projected/3986517e-1d56-42d0-9e97-35f6e18eab05-kube-api-access-hmtkx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:56.042918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:56.042828 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" event={"ID":"36dfbe42-2a12-47c3-80d6-5616a232b401","Type":"ContainerStarted","Data":"b0894df2fe1077e52329f5e23cb652fc29f744c4fab104e5e2709005cdd12555"} Nov 02 20:31:56.042918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:56.042904 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" event={"ID":"36dfbe42-2a12-47c3-80d6-5616a232b401","Type":"ContainerStarted","Data":"d979f9608decee38ebbd29c82c7e131cf0910e02a4a142e0284f20a319c2ca71"} Nov 02 20:31:56.044228 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:56.044205 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" Nov 02 20:31:56.044370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:56.044225 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp" event={"ID":"3986517e-1d56-42d0-9e97-35f6e18eab05","Type":"ContainerDied","Data":"3d1ecf90f803049623ec2288e264090c8ba973db3cddaf78be841beefb70812d"} Nov 02 20:31:56.044370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:56.044260 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d1ecf90f803049623ec2288e264090c8ba973db3cddaf78be841beefb70812d" Nov 02 20:31:56.177164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:56.177108 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" podStartSLOduration=2.177093133 podStartE2EDuration="2.177093133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:56.147934443 +0000 UTC m=+2614.664707029" watchObservedRunningTime="2026-11-02 20:31:56.177093133 +0000 UTC m=+2614.693865718" Nov 02 20:31:57.053409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.053363 5800 generic.go:356] "Generic (PLEG): container finished" podID="36dfbe42-2a12-47c3-80d6-5616a232b401" containerID="b0894df2fe1077e52329f5e23cb652fc29f744c4fab104e5e2709005cdd12555" exitCode=0 Nov 02 20:31:57.053858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.053447 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" event={"ID":"36dfbe42-2a12-47c3-80d6-5616a232b401","Type":"ContainerDied","Data":"b0894df2fe1077e52329f5e23cb652fc29f744c4fab104e5e2709005cdd12555"} Nov 02 20:31:57.634703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.634676 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:31:57.720832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.720790 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-socket\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.720863 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-envoy\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.720952 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-token\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-token\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721073 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-socket" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.721185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721131 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-certs\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721247 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istiod-ca-cert\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721297 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-credential-socket\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721352 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-data\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721431 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-kube-api-access-bsnvw\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-kube-api-access-bsnvw\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721432 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-certs" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.721481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721477 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-podinfo\") pod \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\" (UID: \"51b0d508-997e-4a26-9fa6-f02b398ac7ce\") " Nov 02 20:31:57.721701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721625 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-data" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.721811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721762 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-credential-socket" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.721940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.721916 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istiod-ca-cert" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:57.722198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.722174 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.722391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.722201 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.722391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.722216 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.722391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.722230 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.722391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.722243 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.723372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.723346 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-envoy" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:57.723593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.723566 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-token" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:57.724019 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.724001 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-podinfo" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:57.724769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.724742 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-kube-api-access-bsnvw" pod "51b0d508-997e-4a26-9fa6-f02b398ac7ce" (UID: "51b0d508-997e-4a26-9fa6-f02b398ac7ce"). InnerVolumeSpecName "kube-api-access-bsnvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:57.823508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.823469 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsnvw\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-kube-api-access-bsnvw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.823508 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.823501 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.823726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.823518 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:57.823726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:57.823527 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b0d508-997e-4a26-9fa6-f02b398ac7ce-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:58.064656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.064615 5800 generic.go:356] "Generic (PLEG): container finished" podID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerID="8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6" exitCode=0 Nov 02 20:31:58.065092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.064667 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" Nov 02 20:31:58.065092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.064699 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" event={"ID":"51b0d508-997e-4a26-9fa6-f02b398ac7ce","Type":"ContainerDied","Data":"8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6"} Nov 02 20:31:58.065092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.064757 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb" event={"ID":"51b0d508-997e-4a26-9fa6-f02b398ac7ce","Type":"ContainerDied","Data":"e647db3078fb657bd73f3381f8a6d2393308a2c680c1f0ede89cda18f3c55fc9"} Nov 02 20:31:58.065092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.064787 5800 scope.go:122] "RemoveContainer" containerID="8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6" Nov 02 20:31:58.079240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.079219 5800 scope.go:122] "RemoveContainer" containerID="dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9" Nov 02 20:31:58.090175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.090155 5800 scope.go:122] "RemoveContainer" containerID="8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6" Nov 02 20:31:58.090531 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:58.090497 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6\": container with ID starting with 8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6 not found: ID does not exist" containerID="8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6" Nov 02 20:31:58.090610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.090544 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6"} err="failed to get container status \"8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6\": rpc error: code = NotFound desc = could not find container \"8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6\": container with ID starting with 8b1cb5258f125d6e509d5812901926f4007b83776eb8eb3ad9459a322c1ad6d6 not found: ID does not exist" Nov 02 20:31:58.090610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.090568 5800 scope.go:122] "RemoveContainer" containerID="dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9" Nov 02 20:31:58.090861 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:31:58.090839 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9\": container with ID starting with dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9 not found: ID does not exist" containerID="dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9" Nov 02 20:31:58.090922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.090866 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9"} err="failed to get container status \"dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9\": rpc error: code = NotFound desc = could not find container \"dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9\": container with ID starting with dbf854c11da3d5a33cf614dd91e2e39ca9f948644880b0bb2feed58c836f38e9 not found: ID does not exist" Nov 02 20:31:58.204265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.204240 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:58.223498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.223469 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb"] Nov 02 20:31:58.228227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.228198 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7cad76-openshift-default-6cc5bb877-2zkzb"] Nov 02 20:31:58.329708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.329615 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-openshift-ca\") pod \"36dfbe42-2a12-47c3-80d6-5616a232b401\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " Nov 02 20:31:58.329888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.329749 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dfbe42-2a12-47c3-80d6-5616a232b401-kube-api-access-rzl95\" (UniqueName: \"kubernetes.io/projected/36dfbe42-2a12-47c3-80d6-5616a232b401-kube-api-access-rzl95\") pod \"36dfbe42-2a12-47c3-80d6-5616a232b401\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " Nov 02 20:31:58.329888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.329817 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-xks-ca\") pod \"36dfbe42-2a12-47c3-80d6-5616a232b401\" (UID: \"36dfbe42-2a12-47c3-80d6-5616a232b401\") " Nov 02 20:31:58.331007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.330123 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-openshift-ca" pod "36dfbe42-2a12-47c3-80d6-5616a232b401" (UID: "36dfbe42-2a12-47c3-80d6-5616a232b401"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:58.333239 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.333198 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dfbe42-2a12-47c3-80d6-5616a232b401-kube-api-access-rzl95" pod "36dfbe42-2a12-47c3-80d6-5616a232b401" (UID: "36dfbe42-2a12-47c3-80d6-5616a232b401"). InnerVolumeSpecName "kube-api-access-rzl95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:58.333732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.333705 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-xks-ca" pod "36dfbe42-2a12-47c3-80d6-5616a232b401" (UID: "36dfbe42-2a12-47c3-80d6-5616a232b401"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:31:58.344756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.344725 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8"] Nov 02 20:31:58.431074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.431038 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:58.431074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.431071 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzl95\" (UniqueName: \"kubernetes.io/projected/36dfbe42-2a12-47c3-80d6-5616a232b401-kube-api-access-rzl95\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:58.431352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:58.431087 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dfbe42-2a12-47c3-80d6-5616a232b401-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:59.073465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:59.073431 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" Nov 02 20:31:59.073943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:59.073469 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56" event={"ID":"36dfbe42-2a12-47c3-80d6-5616a232b401","Type":"ContainerDied","Data":"d979f9608decee38ebbd29c82c7e131cf0910e02a4a142e0284f20a319c2ca71"} Nov 02 20:31:59.073943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:59.073533 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d979f9608decee38ebbd29c82c7e131cf0910e02a4a142e0284f20a319c2ca71" Nov 02 20:31:59.629071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:31:59.625552 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" path="/var/lib/kubelet/pods/51b0d508-997e-4a26-9fa6-f02b398ac7ce/volumes" Nov 02 20:32:03.345529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.345438 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" podUID="d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" containerName="payload-pre-processing" containerID="cri-o://ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a" gracePeriod=25 Nov 02 20:32:03.607905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.607839 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:32:03.706973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.706939 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-kube-api-access-zxgqz\" (UniqueName: \"kubernetes.io/projected/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-kube-api-access-zxgqz\") pod \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " Nov 02 20:32:03.707177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.707005 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-plugins-config-volume\") pod \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\" (UID: \"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786\") " Nov 02 20:32:03.707418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.707389 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-plugins-config-volume" pod "d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" (UID: "d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:03.709567 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.709536 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-kube-api-access-zxgqz" pod "d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" (UID: "d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786"). InnerVolumeSpecName "kube-api-access-zxgqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:03.808105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.808069 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxgqz\" (UniqueName: \"kubernetes.io/projected/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-kube-api-access-zxgqz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:03.808105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:03.808099 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:04.120746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.120706 5800 generic.go:356] "Generic (PLEG): container finished" podID="d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" containerID="ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a" exitCode=0 Nov 02 20:32:04.120912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.120757 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" Nov 02 20:32:04.120912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.120785 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" event={"ID":"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786","Type":"ContainerDied","Data":"ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a"} Nov 02 20:32:04.120912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.120841 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8" event={"ID":"d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786","Type":"ContainerDied","Data":"727a5a1f27cdee5cf2962a9e7880ae02359d6880a91e423da09caafe5ec7cb7e"} Nov 02 20:32:04.120912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.120864 5800 scope.go:122] "RemoveContainer" containerID="ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a" Nov 02 20:32:04.131086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.131064 5800 scope.go:122] "RemoveContainer" containerID="ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a" Nov 02 20:32:04.131362 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:32:04.131345 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a\": container with ID starting with ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a not found: ID does not exist" containerID="ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a" Nov 02 20:32:04.131409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.131369 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a"} err="failed to get container status \"ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a\": rpc error: code = NotFound desc = could not find container \"ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a\": container with ID starting with ee37eba587416fb9c2159d3c451e1a77c1e8b6c13ec3cd52ad0630554780da8a not found: ID does not exist" Nov 02 20:32:04.274144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.274112 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8"] Nov 02 20:32:04.320803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:04.320769 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fbe1eb-6c5d97fb6c-kzmb8"] Nov 02 20:32:05.622736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:05.622704 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" path="/var/lib/kubelet/pods/d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786/volumes" Nov 02 20:32:14.077733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.077689 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6"] Nov 02 20:32:14.078226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078211 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" containerName="payload-pre-processing" Nov 02 20:32:14.078313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078228 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" containerName="payload-pre-processing" Nov 02 20:32:14.078313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078242 5800 cpu_manager.go:414] "containerMap: removing container" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" Nov 02 20:32:14.078313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078248 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" Nov 02 20:32:14.078313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078275 5800 cpu_manager.go:414] "containerMap: removing container" podUID="36dfbe42-2a12-47c3-80d6-5616a232b401" containerName="revoke-keys" Nov 02 20:32:14.078313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078294 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dfbe42-2a12-47c3-80d6-5616a232b401" containerName="revoke-keys" Nov 02 20:32:14.078313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078303 5800 cpu_manager.go:414] "containerMap: removing container" podUID="3986517e-1d56-42d0-9e97-35f6e18eab05" containerName="revoke-keys" Nov 02 20:32:14.078313 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078308 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3986517e-1d56-42d0-9e97-35f6e18eab05" containerName="revoke-keys" Nov 02 20:32:14.078544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078322 5800 cpu_manager.go:414] "containerMap: removing container" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" Nov 02 20:32:14.078544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078327 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" Nov 02 20:32:14.078544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078411 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dfbe42-2a12-47c3-80d6-5616a232b401" containerName="revoke-keys" Nov 02 20:32:14.078544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078421 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="3986517e-1d56-42d0-9e97-35f6e18eab05" containerName="revoke-keys" Nov 02 20:32:14.078544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078428 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" Nov 02 20:32:14.078544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078435 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aa21ac-a13d-43c9-a9ad-2f4bf0cb8786" containerName="payload-pre-processing" Nov 02 20:32:14.078544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.078441 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b0d508-997e-4a26-9fa6-f02b398ac7ce" containerName="istio-proxy" Nov 02 20:32:14.083120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.083102 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.117189 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.117157 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcl9\" (UniqueName: \"kubernetes.io/projected/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-kube-api-access-4rcl9\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.117346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.117208 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.117407 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.117379 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.135025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.134995 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6"] Nov 02 20:32:14.177395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.177366 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:32:14.177578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.177404 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:32:14.178796 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.178775 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:14.178953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.178890 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:32:14.218832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.218802 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.219012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.218868 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcl9\" (UniqueName: \"kubernetes.io/projected/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-kube-api-access-4rcl9\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.219012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.218898 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.219665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.219628 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.221202 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.221180 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.272220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.272182 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcl9\" (UniqueName: \"kubernetes.io/projected/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-kube-api-access-4rcl9\") pod \"maas-api-revoke-sub-d8be8ab5-9j2q6\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.396771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.396687 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:14.776579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:14.776548 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6"] Nov 02 20:32:14.777410 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:32:14.777386 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cfe6c7_ef8e_4fae_b698_68bc4ce67aa5.slice/crio-1faf4914872f8533cee09b79ce236a78f916b15e0c818c4777860390c465f367 WatchSource:0}: Error finding container 1faf4914872f8533cee09b79ce236a78f916b15e0c818c4777860390c465f367: Status 404 returned error can't find the container with id 1faf4914872f8533cee09b79ce236a78f916b15e0c818c4777860390c465f367 Nov 02 20:32:15.215045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.214983 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" event={"ID":"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5","Type":"ContainerStarted","Data":"1b0ba8ce401422ff1f4e531ad4e03c954eddd4dfbd47e3dcfe20c0bc7ed05b1c"} Nov 02 20:32:15.215592 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.215101 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" event={"ID":"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5","Type":"ContainerStarted","Data":"1faf4914872f8533cee09b79ce236a78f916b15e0c818c4777860390c465f367"} Nov 02 20:32:15.324534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.324428 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" podStartSLOduration=1.324413056 podStartE2EDuration="1.324413056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:15.322387616 +0000 UTC m=+2633.839160201" watchObservedRunningTime="2026-11-02 20:32:15.324413056 +0000 UTC m=+2633.841185638" Nov 02 20:32:15.795574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.795535 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp"] Nov 02 20:32:15.799651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.799633 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.838423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.838380 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.838626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.838565 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvt7\" (UniqueName: \"kubernetes.io/projected/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-kube-api-access-hpvt7\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.838703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.838686 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.853612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.853581 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp"] Nov 02 20:32:15.940121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.940029 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvt7\" (UniqueName: \"kubernetes.io/projected/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-kube-api-access-hpvt7\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.940346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.940135 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.940346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.940214 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.941166 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.941123 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.942899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.942866 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:15.993221 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:15.993186 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvt7\" (UniqueName: \"kubernetes.io/projected/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-kube-api-access-hpvt7\") pod \"maas-api-revoke-sub-e07aca44-hw4wp\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:16.112958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:16.112921 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:16.223504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:16.223415 5800 generic.go:356] "Generic (PLEG): container finished" podID="f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" containerID="1b0ba8ce401422ff1f4e531ad4e03c954eddd4dfbd47e3dcfe20c0bc7ed05b1c" exitCode=0 Nov 02 20:32:16.223866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:16.223500 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" event={"ID":"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5","Type":"ContainerDied","Data":"1b0ba8ce401422ff1f4e531ad4e03c954eddd4dfbd47e3dcfe20c0bc7ed05b1c"} Nov 02 20:32:16.486208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:16.486181 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp"] Nov 02 20:32:16.487256 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:32:16.487229 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbdea1c_7ae8_41e2_aec0_53d15f94b738.slice/crio-d35671840556ba25639888f0264b0399064ca58c6fa5a424da7a7cbd648b1897 WatchSource:0}: Error finding container d35671840556ba25639888f0264b0399064ca58c6fa5a424da7a7cbd648b1897: Status 404 returned error can't find the container with id d35671840556ba25639888f0264b0399064ca58c6fa5a424da7a7cbd648b1897 Nov 02 20:32:17.269516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.269468 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" event={"ID":"7dbdea1c-7ae8-41e2-aec0-53d15f94b738","Type":"ContainerStarted","Data":"f4b63b4a72514802a822e69ab6090ad5220aacc724d9387326a32b3a6efa0a6b"} Nov 02 20:32:17.270048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.269531 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" event={"ID":"7dbdea1c-7ae8-41e2-aec0-53d15f94b738","Type":"ContainerStarted","Data":"d35671840556ba25639888f0264b0399064ca58c6fa5a424da7a7cbd648b1897"} Nov 02 20:32:17.378747 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.378577 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" podStartSLOduration=2.378557152 podStartE2EDuration="2.378557152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:17.375667376 +0000 UTC m=+2635.892439965" watchObservedRunningTime="2026-11-02 20:32:17.378557152 +0000 UTC m=+2635.895329738" Nov 02 20:32:17.461628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.461596 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:17.570212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.569982 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-kube-api-access-4rcl9\" (UniqueName: \"kubernetes.io/projected/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-kube-api-access-4rcl9\") pod \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " Nov 02 20:32:17.570542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.570517 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-openshift-ca\") pod \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " Nov 02 20:32:17.570698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.570682 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-xks-ca\") pod \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\" (UID: \"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5\") " Nov 02 20:32:17.577716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.577654 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-openshift-ca" pod "f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" (UID: "f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:17.596045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.595991 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-xks-ca" pod "f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" (UID: "f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:17.596332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.596275 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-kube-api-access-4rcl9" pod "f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" (UID: "f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5"). InnerVolumeSpecName "kube-api-access-4rcl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:17.671894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.671854 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rcl9\" (UniqueName: \"kubernetes.io/projected/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-kube-api-access-4rcl9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.671894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.671884 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:17.671894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:17.671898 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:18.276213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:18.276171 5800 generic.go:356] "Generic (PLEG): container finished" podID="7dbdea1c-7ae8-41e2-aec0-53d15f94b738" containerID="f4b63b4a72514802a822e69ab6090ad5220aacc724d9387326a32b3a6efa0a6b" exitCode=0 Nov 02 20:32:18.276213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:18.276204 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" event={"ID":"7dbdea1c-7ae8-41e2-aec0-53d15f94b738","Type":"ContainerDied","Data":"f4b63b4a72514802a822e69ab6090ad5220aacc724d9387326a32b3a6efa0a6b"} Nov 02 20:32:18.277780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:18.277761 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" Nov 02 20:32:18.277884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:18.277787 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6" event={"ID":"f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5","Type":"ContainerDied","Data":"1faf4914872f8533cee09b79ce236a78f916b15e0c818c4777860390c465f367"} Nov 02 20:32:18.277884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:18.277832 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1faf4914872f8533cee09b79ce236a78f916b15e0c818c4777860390c465f367" Nov 02 20:32:19.427587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.427553 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:19.492980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.492948 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-openshift-ca\") pod \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " Nov 02 20:32:19.493158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.493063 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-xks-ca\") pod \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " Nov 02 20:32:19.493158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.493116 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-kube-api-access-hpvt7\" (UniqueName: \"kubernetes.io/projected/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-kube-api-access-hpvt7\") pod \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\" (UID: \"7dbdea1c-7ae8-41e2-aec0-53d15f94b738\") " Nov 02 20:32:19.493328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.493302 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-openshift-ca" pod "7dbdea1c-7ae8-41e2-aec0-53d15f94b738" (UID: "7dbdea1c-7ae8-41e2-aec0-53d15f94b738"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:19.493418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.493399 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:19.495138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.495109 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-xks-ca" pod "7dbdea1c-7ae8-41e2-aec0-53d15f94b738" (UID: "7dbdea1c-7ae8-41e2-aec0-53d15f94b738"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:19.495321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.495300 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-kube-api-access-hpvt7" pod "7dbdea1c-7ae8-41e2-aec0-53d15f94b738" (UID: "7dbdea1c-7ae8-41e2-aec0-53d15f94b738"). InnerVolumeSpecName "kube-api-access-hpvt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:19.594649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.594565 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:19.594649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:19.594598 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvt7\" (UniqueName: \"kubernetes.io/projected/7dbdea1c-7ae8-41e2-aec0-53d15f94b738-kube-api-access-hpvt7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:20.292015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:20.291988 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" Nov 02 20:32:20.292015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:20.292001 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp" event={"ID":"7dbdea1c-7ae8-41e2-aec0-53d15f94b738","Type":"ContainerDied","Data":"d35671840556ba25639888f0264b0399064ca58c6fa5a424da7a7cbd648b1897"} Nov 02 20:32:20.292233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:20.292047 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35671840556ba25639888f0264b0399064ca58c6fa5a424da7a7cbd648b1897" Nov 02 20:32:22.696713 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:32:22.696677 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:32:30.982681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.982645 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl"] Nov 02 20:32:30.983211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.983147 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" containerName="revoke-keys" Nov 02 20:32:30.983211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.983157 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" containerName="revoke-keys" Nov 02 20:32:30.983211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.983195 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7dbdea1c-7ae8-41e2-aec0-53d15f94b738" containerName="revoke-keys" Nov 02 20:32:30.983211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.983202 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbdea1c-7ae8-41e2-aec0-53d15f94b738" containerName="revoke-keys" Nov 02 20:32:30.983380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.983317 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbdea1c-7ae8-41e2-aec0-53d15f94b738" containerName="revoke-keys" Nov 02 20:32:30.983380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.983329 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" containerName="revoke-keys" Nov 02 20:32:30.988410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:30.988391 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.081409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.081374 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl"] Nov 02 20:32:31.083730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.083700 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-14d090-openshift-default-dockercfg-5wl9l\"" Nov 02 20:32:31.125169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125134 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125192 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125326 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125373 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjhg\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-kube-api-access-qzjhg\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125399 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125547 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125613 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125667 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.125830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.125773 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa138c-489b-4b82-99fa-329c35ddecc7-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.227505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227463 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.227726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227564 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.227726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227591 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.227726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227658 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa138c-489b-4b82-99fa-329c35ddecc7-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.227726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227695 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227732 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227769 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227817 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjhg\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-kube-api-access-qzjhg\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227846 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.227943 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-credential-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.228068 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-socket\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.228765 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-data\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.228780 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-certs\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.228984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.228891 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa138c-489b-4b82-99fa-329c35ddecc7-istiod-ca-cert\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.230728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.230700 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-envoy\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.230929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.230909 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-podinfo\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.281777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.281740 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-token\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.281942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.281918 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjhg\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-kube-api-access-qzjhg\") pod \"e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.301558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.301525 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:31.723457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.723423 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl"] Nov 02 20:32:31.723803 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:32:31.723775 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffaa138c_489b_4b82_99fa_329c35ddecc7.slice/crio-fff5190488fd2b6f3f243344677e13180a3acf2cb026598f78c6f8532b32f159 WatchSource:0}: Error finding container fff5190488fd2b6f3f243344677e13180a3acf2cb026598f78c6f8532b32f159: Status 404 returned error can't find the container with id fff5190488fd2b6f3f243344677e13180a3acf2cb026598f78c6f8532b32f159 Nov 02 20:32:31.726405 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.726371 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:31.726523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.726450 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:31.726523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:31.726499 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:32.377677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.377638 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" event={"ID":"ffaa138c-489b-4b82-99fa-329c35ddecc7","Type":"ContainerStarted","Data":"de1279d8d67bb7c115af1a5ca9650fbcae5badea73fb8dd1f85a9d9641189260"} Nov 02 20:32:32.378179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.377700 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" event={"ID":"ffaa138c-489b-4b82-99fa-329c35ddecc7","Type":"ContainerStarted","Data":"fff5190488fd2b6f3f243344677e13180a3acf2cb026598f78c6f8532b32f159"} Nov 02 20:32:32.492567 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.492497 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podStartSLOduration=2.4924719140000002 podStartE2EDuration="2.492471914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:32.486587331 +0000 UTC m=+2651.003359918" watchObservedRunningTime="2026-11-02 20:32:32.492471914 +0000 UTC m=+2651.009244501" Nov 02 20:32:32.810551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.810513 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d"] Nov 02 20:32:32.814833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.814810 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:32.889310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.873648 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:32:32.909623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.892096 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:32.909623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.895150 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:32:32.909623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.895243 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:32:32.909623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.895502 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d"] Nov 02 20:32:32.909623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.907875 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:32.909623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.909401 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpwb\" (UniqueName: \"kubernetes.io/projected/6be2f913-18b0-439f-ad42-9485d5eeed29-kube-api-access-5rpwb\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:32.909623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:32.909483 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.010693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.010641 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpwb\" (UniqueName: \"kubernetes.io/projected/6be2f913-18b0-439f-ad42-9485d5eeed29-kube-api-access-5rpwb\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.011021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.010982 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.011247 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.011230 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.012199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.012170 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.055304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.052844 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.087347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.084827 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpwb\" (UniqueName: \"kubernetes.io/projected/6be2f913-18b0-439f-ad42-9485d5eeed29-kube-api-access-5rpwb\") pod \"maas-api-revoke-sub-13636581-ngr4d\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.148305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.145511 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:33.302465 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.301822 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:33.304311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.303630 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 20:32:33.304311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.303685 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 20:32:33.306680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.306112 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm"] Nov 02 20:32:33.311851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.311827 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:33.373593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.359765 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm"] Nov 02 20:32:33.373593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.361050 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-920071\"" Nov 02 20:32:33.373593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.361170 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-920071-dockercfg-4827s\"" Nov 02 20:32:33.418353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.418264 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa663b4-496d-4283-8fc8-5d48c9c47913-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:33.418826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.418362 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hrt\" (UniqueName: \"kubernetes.io/projected/dfa663b4-496d-4283-8fc8-5d48c9c47913-kube-api-access-c6hrt\") pod \"payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:33.519761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.519724 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa663b4-496d-4283-8fc8-5d48c9c47913-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:33.519929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.519781 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hrt\" (UniqueName: \"kubernetes.io/projected/dfa663b4-496d-4283-8fc8-5d48c9c47913-kube-api-access-c6hrt\") pod \"payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:33.520513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.520494 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa663b4-496d-4283-8fc8-5d48c9c47913-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:33.572917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.572878 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hrt\" (UniqueName: \"kubernetes.io/projected/dfa663b4-496d-4283-8fc8-5d48c9c47913-kube-api-access-c6hrt\") pod \"payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:33.618831 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:32:33.618195 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be2f913_18b0_439f_ad42_9485d5eeed29.slice/crio-a35c0c24dc978fd516151857f4ad805cf66f65123234bd34af47168d3b52a0e6 WatchSource:0}: Error finding container a35c0c24dc978fd516151857f4ad805cf66f65123234bd34af47168d3b52a0e6: Status 404 returned error can't find the container with id a35c0c24dc978fd516151857f4ad805cf66f65123234bd34af47168d3b52a0e6 Nov 02 20:32:33.624198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.624142 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d"] Nov 02 20:32:33.683414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:33.682997 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:34.093366 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.093335 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm"] Nov 02 20:32:34.094376 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:32:34.094352 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa663b4_496d_4283_8fc8_5d48c9c47913.slice/crio-14969ede112dc12701e5b5cd0c528029d9555a184ce6cb10b17fc2db9f55f3e9 WatchSource:0}: Error finding container 14969ede112dc12701e5b5cd0c528029d9555a184ce6cb10b17fc2db9f55f3e9: Status 404 returned error can't find the container with id 14969ede112dc12701e5b5cd0c528029d9555a184ce6cb10b17fc2db9f55f3e9 Nov 02 20:32:34.302380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.302346 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 20:32:34.302559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.302399 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 20:32:34.394516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.394473 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" event={"ID":"dfa663b4-496d-4283-8fc8-5d48c9c47913","Type":"ContainerStarted","Data":"10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab"} Nov 02 20:32:34.394705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.394542 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" event={"ID":"dfa663b4-496d-4283-8fc8-5d48c9c47913","Type":"ContainerStarted","Data":"14969ede112dc12701e5b5cd0c528029d9555a184ce6cb10b17fc2db9f55f3e9"} Nov 02 20:32:34.394705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.394658 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:34.396149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.396123 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" event={"ID":"6be2f913-18b0-439f-ad42-9485d5eeed29","Type":"ContainerStarted","Data":"4daaf79403c3ba73dc8aeecea31d20b2b27328800f4b0bbb329d9122c1f0ea98"} Nov 02 20:32:34.396244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.396178 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" event={"ID":"6be2f913-18b0-439f-ad42-9485d5eeed29","Type":"ContainerStarted","Data":"a35c0c24dc978fd516151857f4ad805cf66f65123234bd34af47168d3b52a0e6"} Nov 02 20:32:34.505699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.505648 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" podStartSLOduration=1.505633204 podStartE2EDuration="1.505633204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:34.504149952 +0000 UTC m=+2653.020922538" watchObservedRunningTime="2026-11-02 20:32:34.505633204 +0000 UTC m=+2653.022405785" Nov 02 20:32:34.669308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:34.665621 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" podStartSLOduration=2.665602297 podStartE2EDuration="2.665602297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:34.61510328 +0000 UTC m=+2653.131875869" watchObservedRunningTime="2026-11-02 20:32:34.665602297 +0000 UTC m=+2653.182375079" Nov 02 20:32:35.302302 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:35.302241 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 20:32:35.302500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:35.302324 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 20:32:35.405033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:35.404995 5800 generic.go:356] "Generic (PLEG): container finished" podID="6be2f913-18b0-439f-ad42-9485d5eeed29" containerID="4daaf79403c3ba73dc8aeecea31d20b2b27328800f4b0bbb329d9122c1f0ea98" exitCode=0 Nov 02 20:32:35.405257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:35.405094 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" event={"ID":"6be2f913-18b0-439f-ad42-9485d5eeed29","Type":"ContainerDied","Data":"4daaf79403c3ba73dc8aeecea31d20b2b27328800f4b0bbb329d9122c1f0ea98"} Nov 02 20:32:36.302066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.302031 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 20:32:36.302520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.302098 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 20:32:36.618158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.618128 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:36.683532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.683493 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-openshift-ca\") pod \"6be2f913-18b0-439f-ad42-9485d5eeed29\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " Nov 02 20:32:36.683694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.683620 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be2f913-18b0-439f-ad42-9485d5eeed29-kube-api-access-5rpwb\" (UniqueName: \"kubernetes.io/projected/6be2f913-18b0-439f-ad42-9485d5eeed29-kube-api-access-5rpwb\") pod \"6be2f913-18b0-439f-ad42-9485d5eeed29\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " Nov 02 20:32:36.683694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.683657 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-xks-ca\") pod \"6be2f913-18b0-439f-ad42-9485d5eeed29\" (UID: \"6be2f913-18b0-439f-ad42-9485d5eeed29\") " Nov 02 20:32:36.684250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.683882 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-openshift-ca" pod "6be2f913-18b0-439f-ad42-9485d5eeed29" (UID: "6be2f913-18b0-439f-ad42-9485d5eeed29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:36.684250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.684087 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:36.695915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.695876 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be2f913-18b0-439f-ad42-9485d5eeed29-kube-api-access-5rpwb" pod "6be2f913-18b0-439f-ad42-9485d5eeed29" (UID: "6be2f913-18b0-439f-ad42-9485d5eeed29"). InnerVolumeSpecName "kube-api-access-5rpwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:36.696070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.695919 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-xks-ca" pod "6be2f913-18b0-439f-ad42-9485d5eeed29" (UID: "6be2f913-18b0-439f-ad42-9485d5eeed29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:36.785077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.785036 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rpwb\" (UniqueName: \"kubernetes.io/projected/6be2f913-18b0-439f-ad42-9485d5eeed29-kube-api-access-5rpwb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:36.785077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:36.785075 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be2f913-18b0-439f-ad42-9485d5eeed29-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:37.302841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.302802 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 20:32:37.303259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.302884 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 20:32:37.421000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.420964 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" event={"ID":"6be2f913-18b0-439f-ad42-9485d5eeed29","Type":"ContainerDied","Data":"a35c0c24dc978fd516151857f4ad805cf66f65123234bd34af47168d3b52a0e6"} Nov 02 20:32:37.421178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.421050 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35c0c24dc978fd516151857f4ad805cf66f65123234bd34af47168d3b52a0e6" Nov 02 20:32:37.421178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.420972 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d" Nov 02 20:32:37.697528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.697441 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt"] Nov 02 20:32:37.698006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.697991 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6be2f913-18b0-439f-ad42-9485d5eeed29" containerName="revoke-keys" Nov 02 20:32:37.698054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.698008 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be2f913-18b0-439f-ad42-9485d5eeed29" containerName="revoke-keys" Nov 02 20:32:37.698147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.698137 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be2f913-18b0-439f-ad42-9485d5eeed29" containerName="revoke-keys" Nov 02 20:32:37.701773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.701749 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.767307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.757819 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:32:37.767307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.757895 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:37.767307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.763750 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt"] Nov 02 20:32:37.795353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.795310 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.795555 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.795368 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.795625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.795602 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9cl\" (UniqueName: \"kubernetes.io/projected/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-kube-api-access-rd9cl\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.796299 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.796259 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:32:37.796435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.796408 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:32:37.897141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.897107 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9cl\" (UniqueName: \"kubernetes.io/projected/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-kube-api-access-rd9cl\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.897370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.897171 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.897370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.897203 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.898048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.898026 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.899668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.899648 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:37.950764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:37.950684 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9cl\" (UniqueName: \"kubernetes.io/projected/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-kube-api-access-rd9cl\") pod \"maas-api-revoke-sub-0e5da59f-ps8vt\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:38.017569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:38.017535 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:38.302392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:38.302353 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 02 20:32:38.302618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:38.302422 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 02 20:32:38.387930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:38.387894 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt"] Nov 02 20:32:38.388805 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:32:38.388771 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4a223f_8f4b_4e50_8a2d_07dc52b0df3c.slice/crio-e0c64a22653bb69d1d9c33c275b69739787716a2011d152b97dba71f551b8c20 WatchSource:0}: Error finding container e0c64a22653bb69d1d9c33c275b69739787716a2011d152b97dba71f551b8c20: Status 404 returned error can't find the container with id e0c64a22653bb69d1d9c33c275b69739787716a2011d152b97dba71f551b8c20 Nov 02 20:32:38.432194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:38.432157 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" event={"ID":"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c","Type":"ContainerStarted","Data":"e0c64a22653bb69d1d9c33c275b69739787716a2011d152b97dba71f551b8c20"} Nov 02 20:32:39.443232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:39.443189 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" event={"ID":"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c","Type":"ContainerStarted","Data":"1fe05f367f252cba99ac701d3a5197819281faa5fa295e3b24bc459fda49cdf8"} Nov 02 20:32:39.550517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:39.550467 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" podStartSLOduration=2.550450919 podStartE2EDuration="2.550450919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:39.548301311 +0000 UTC m=+2658.065073889" watchObservedRunningTime="2026-11-02 20:32:39.550450919 +0000 UTC m=+2658.067223505" Nov 02 20:32:39.572443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:39.572406 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:39.572810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:39.572779 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:39.573931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:39.573751 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:32:40.453343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:40.453272 5800 generic.go:356] "Generic (PLEG): container finished" podID="ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" containerID="1fe05f367f252cba99ac701d3a5197819281faa5fa295e3b24bc459fda49cdf8" exitCode=0 Nov 02 20:32:40.453777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:40.453309 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" event={"ID":"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c","Type":"ContainerDied","Data":"1fe05f367f252cba99ac701d3a5197819281faa5fa295e3b24bc459fda49cdf8"} Nov 02 20:32:41.619477 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.619444 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:41.748071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.744164 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-kube-api-access-rd9cl\" (UniqueName: \"kubernetes.io/projected/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-kube-api-access-rd9cl\") pod \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " Nov 02 20:32:41.748071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.744277 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-openshift-ca\") pod \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " Nov 02 20:32:41.748071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.744438 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-xks-ca\") pod \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\" (UID: \"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c\") " Nov 02 20:32:41.748071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.745556 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-openshift-ca" pod "ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" (UID: "ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:41.748071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.747137 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-kube-api-access-rd9cl" pod "ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" (UID: "ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c"). InnerVolumeSpecName "kube-api-access-rd9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:41.750263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.750225 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-xks-ca" pod "ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" (UID: "ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:41.846486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.846447 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:41.846486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.846486 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:41.846700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:41.846503 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9cl\" (UniqueName: \"kubernetes.io/projected/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c-kube-api-access-rd9cl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:42.470094 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:42.470052 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" Nov 02 20:32:42.470304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:42.470079 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt" event={"ID":"ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c","Type":"ContainerDied","Data":"e0c64a22653bb69d1d9c33c275b69739787716a2011d152b97dba71f551b8c20"} Nov 02 20:32:42.470304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:42.470149 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c64a22653bb69d1d9c33c275b69739787716a2011d152b97dba71f551b8c20" Nov 02 20:32:45.408740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:45.408713 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:32:52.697187 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:32:52.697142 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:32:53.088795 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.088752 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs"] Nov 02 20:32:53.089390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.089369 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" containerName="revoke-keys" Nov 02 20:32:53.089390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.089390 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" containerName="revoke-keys" Nov 02 20:32:53.089571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.089547 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" containerName="revoke-keys" Nov 02 20:32:53.093114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.093093 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.142277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.142247 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:32:53.142524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.142318 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:32:53.142524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.142374 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:32:53.142524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.142401 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:32:53.144395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.144371 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs"] Nov 02 20:32:53.183038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.183005 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.183229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.183156 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.183323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.183259 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhwr\" (UniqueName: \"kubernetes.io/projected/d41a78d3-5375-4d5e-9746-bd27eefc0c27-kube-api-access-8hhwr\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.284186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.284154 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.284397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.284237 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.284397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.284332 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhwr\" (UniqueName: \"kubernetes.io/projected/d41a78d3-5375-4d5e-9746-bd27eefc0c27-kube-api-access-8hhwr\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.284997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.284941 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.286653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.286628 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.337084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.337045 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhwr\" (UniqueName: \"kubernetes.io/projected/d41a78d3-5375-4d5e-9746-bd27eefc0c27-kube-api-access-8hhwr\") pod \"maas-api-revoke-sub-d15d4853-qdrcs\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.406868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.406788 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:53.783681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:53.783649 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs"] Nov 02 20:32:53.784804 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:32:53.784778 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41a78d3_5375_4d5e_9746_bd27eefc0c27.slice/crio-59df63ec25172e4556933b682088e19d596516c97e7599a34d3f7fd57371392a WatchSource:0}: Error finding container 59df63ec25172e4556933b682088e19d596516c97e7599a34d3f7fd57371392a: Status 404 returned error can't find the container with id 59df63ec25172e4556933b682088e19d596516c97e7599a34d3f7fd57371392a Nov 02 20:32:54.562849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:54.562805 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" event={"ID":"d41a78d3-5375-4d5e-9746-bd27eefc0c27","Type":"ContainerStarted","Data":"dd4f550f9b02c9db26a4ea9a5c08f5bbc32e9b32cffc0a44f0c313f99223b6ec"} Nov 02 20:32:54.563049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:54.562868 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" event={"ID":"d41a78d3-5375-4d5e-9746-bd27eefc0c27","Type":"ContainerStarted","Data":"59df63ec25172e4556933b682088e19d596516c97e7599a34d3f7fd57371392a"} Nov 02 20:32:54.671403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:54.671342 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" podStartSLOduration=1.671323117 podStartE2EDuration="1.671323117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:54.668056352 +0000 UTC m=+2673.184828937" watchObservedRunningTime="2026-11-02 20:32:54.671323117 +0000 UTC m=+2673.188095704" Nov 02 20:32:55.573501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:55.573458 5800 generic.go:356] "Generic (PLEG): container finished" podID="d41a78d3-5375-4d5e-9746-bd27eefc0c27" containerID="dd4f550f9b02c9db26a4ea9a5c08f5bbc32e9b32cffc0a44f0c313f99223b6ec" exitCode=0 Nov 02 20:32:55.573941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:55.573523 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" event={"ID":"d41a78d3-5375-4d5e-9746-bd27eefc0c27","Type":"ContainerDied","Data":"dd4f550f9b02c9db26a4ea9a5c08f5bbc32e9b32cffc0a44f0c313f99223b6ec"} Nov 02 20:32:55.772540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:55.772504 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh"] Nov 02 20:32:55.772827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:55.772789 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" containerID="cri-o://7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56" gracePeriod=30 Nov 02 20:32:55.985698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:55.985621 5800 patch_prober.go:29] interesting pod/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:32:55.985858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:55.985692 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:32:56.736964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.736936 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:56.847746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.847668 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41a78d3-5375-4d5e-9746-bd27eefc0c27-kube-api-access-8hhwr\" (UniqueName: \"kubernetes.io/projected/d41a78d3-5375-4d5e-9746-bd27eefc0c27-kube-api-access-8hhwr\") pod \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " Nov 02 20:32:56.847746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.847707 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-xks-ca\") pod \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " Nov 02 20:32:56.847981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.847835 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-openshift-ca\") pod \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\" (UID: \"d41a78d3-5375-4d5e-9746-bd27eefc0c27\") " Nov 02 20:32:56.848409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.848379 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-openshift-ca" pod "d41a78d3-5375-4d5e-9746-bd27eefc0c27" (UID: "d41a78d3-5375-4d5e-9746-bd27eefc0c27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:32:56.850794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.850497 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41a78d3-5375-4d5e-9746-bd27eefc0c27-kube-api-access-8hhwr" pod "d41a78d3-5375-4d5e-9746-bd27eefc0c27" (UID: "d41a78d3-5375-4d5e-9746-bd27eefc0c27"). InnerVolumeSpecName "kube-api-access-8hhwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:32:56.850794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.850580 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-xks-ca" pod "d41a78d3-5375-4d5e-9746-bd27eefc0c27" (UID: "d41a78d3-5375-4d5e-9746-bd27eefc0c27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:32:56.949167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.949129 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhwr\" (UniqueName: \"kubernetes.io/projected/d41a78d3-5375-4d5e-9746-bd27eefc0c27-kube-api-access-8hhwr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:56.949167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.949163 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:56.949167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:56.949173 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41a78d3-5375-4d5e-9746-bd27eefc0c27-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:32:57.589554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:57.589521 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" event={"ID":"d41a78d3-5375-4d5e-9746-bd27eefc0c27","Type":"ContainerDied","Data":"59df63ec25172e4556933b682088e19d596516c97e7599a34d3f7fd57371392a"} Nov 02 20:32:57.589554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:57.589556 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs" Nov 02 20:32:57.589779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:57.589573 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59df63ec25172e4556933b682088e19d596516c97e7599a34d3f7fd57371392a" Nov 02 20:32:57.912520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:57.912425 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb"] Nov 02 20:32:58.014392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:32:58.014357 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz"] Nov 02 20:33:01.055827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.055794 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:33:01.110787 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.110750 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-kube-api-access-j79tq\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-kube-api-access-j79tq\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.110983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.110828 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-podinfo\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.110983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.110871 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-socket\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.110983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.110925 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-envoy\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.110983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.110952 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-token\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-token\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.111212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111017 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c337afa1-a2b2-47fc-accb-4e11c7251b46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c337afa1-a2b2-47fc-accb-4e11c7251b46-istiod-ca-cert\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.111212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111071 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-certs\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.111212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111131 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-data\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.111212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111199 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-credential-socket\") pod \"c337afa1-a2b2-47fc-accb-4e11c7251b46\" (UID: \"c337afa1-a2b2-47fc-accb-4e11c7251b46\") " Nov 02 20:33:01.111433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111230 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-socket" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:01.111494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111423 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-certs" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:01.111550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111513 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c337afa1-a2b2-47fc-accb-4e11c7251b46-istiod-ca-cert" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:01.111672 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111653 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-credential-socket" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:01.111761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111732 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-data" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:01.111835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111790 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.111835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111811 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c337afa1-a2b2-47fc-accb-4e11c7251b46-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.111835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111826 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.111997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.111841 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.113464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.113429 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-envoy" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:01.113623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.113550 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-podinfo" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:33:01.113873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.113856 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-kube-api-access-j79tq" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "kube-api-access-j79tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:01.114320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.114299 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-token" pod "c337afa1-a2b2-47fc-accb-4e11c7251b46" (UID: "c337afa1-a2b2-47fc-accb-4e11c7251b46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:01.212882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.212765 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.212882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.212803 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.212882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.212817 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.212882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.212831 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j79tq\" (UniqueName: \"kubernetes.io/projected/c337afa1-a2b2-47fc-accb-4e11c7251b46-kube-api-access-j79tq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.212882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.212848 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c337afa1-a2b2-47fc-accb-4e11c7251b46-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:01.624443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.624398 5800 generic.go:356] "Generic (PLEG): container finished" podID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerID="7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56" exitCode=0 Nov 02 20:33:01.624627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.624514 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" Nov 02 20:33:01.625419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.625396 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" event={"ID":"c337afa1-a2b2-47fc-accb-4e11c7251b46","Type":"ContainerDied","Data":"7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56"} Nov 02 20:33:01.625541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.625440 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh" event={"ID":"c337afa1-a2b2-47fc-accb-4e11c7251b46","Type":"ContainerDied","Data":"fdc15e861ad396b212c80b3a97693c2bb5553a1dc6747b81fc75fca370660536"} Nov 02 20:33:01.625541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.625466 5800 scope.go:122] "RemoveContainer" containerID="7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56" Nov 02 20:33:01.644055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.643928 5800 scope.go:122] "RemoveContainer" containerID="012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f" Nov 02 20:33:01.655501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.655479 5800 scope.go:122] "RemoveContainer" containerID="7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56" Nov 02 20:33:01.655868 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:01.655844 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56\": container with ID starting with 7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56 not found: ID does not exist" containerID="7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56" Nov 02 20:33:01.655942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.655878 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56"} err="failed to get container status \"7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56\": rpc error: code = NotFound desc = could not find container \"7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56\": container with ID starting with 7bfaa2ccbd0d86d400ce322639dcdce30f0aa60e472546d1b584be47d4904b56 not found: ID does not exist" Nov 02 20:33:01.655942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.655897 5800 scope.go:122] "RemoveContainer" containerID="012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f" Nov 02 20:33:01.656235 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:01.656202 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f\": container with ID starting with 012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f not found: ID does not exist" containerID="012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f" Nov 02 20:33:01.656378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.656239 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f"} err="failed to get container status \"012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f\": rpc error: code = NotFound desc = could not find container \"012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f\": container with ID starting with 012f60a8b28528337b5285075dba62db8b05afc3e5d87b60f7fa85326995742f not found: ID does not exist" Nov 02 20:33:01.786198 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.786162 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh"] Nov 02 20:33:01.833117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:01.833072 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fbe1eb-openshift-default-78c85c469-l7bsh"] Nov 02 20:33:02.913208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:02.913134 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" podUID="d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" containerName="payload-processing" containerID="cri-o://c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809" gracePeriod=25 Nov 02 20:33:03.014670 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.014629 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" podUID="68a614a3-3987-4614-a716-d188df5b383c" containerName="payload-pre-processing" containerID="cri-o://65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495" gracePeriod=25 Nov 02 20:33:03.193959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.193926 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:33:03.329640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.329597 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:33:03.338154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.338116 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-plugins-config-volume\") pod \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " Nov 02 20:33:03.338375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.338222 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-kube-api-access-7mmtw\" (UniqueName: \"kubernetes.io/projected/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-kube-api-access-7mmtw\") pod \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\" (UID: \"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e\") " Nov 02 20:33:03.338625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.338598 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-plugins-config-volume" pod "d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" (UID: "d5b30d57-44d0-42a7-b69d-2fa8cc1a320e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:03.340653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.340624 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-kube-api-access-7mmtw" pod "d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" (UID: "d5b30d57-44d0-42a7-b69d-2fa8cc1a320e"). InnerVolumeSpecName "kube-api-access-7mmtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:03.439764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.439676 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a614a3-3987-4614-a716-d188df5b383c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a614a3-3987-4614-a716-d188df5b383c-plugins-config-volume\") pod \"68a614a3-3987-4614-a716-d188df5b383c\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " Nov 02 20:33:03.439948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.439769 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a614a3-3987-4614-a716-d188df5b383c-kube-api-access-75z57\" (UniqueName: \"kubernetes.io/projected/68a614a3-3987-4614-a716-d188df5b383c-kube-api-access-75z57\") pod \"68a614a3-3987-4614-a716-d188df5b383c\" (UID: \"68a614a3-3987-4614-a716-d188df5b383c\") " Nov 02 20:33:03.440491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.440187 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmtw\" (UniqueName: \"kubernetes.io/projected/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-kube-api-access-7mmtw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.440491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.440215 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.440491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.440451 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a614a3-3987-4614-a716-d188df5b383c-plugins-config-volume" pod "68a614a3-3987-4614-a716-d188df5b383c" (UID: "68a614a3-3987-4614-a716-d188df5b383c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:03.442794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.442763 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a614a3-3987-4614-a716-d188df5b383c-kube-api-access-75z57" pod "68a614a3-3987-4614-a716-d188df5b383c" (UID: "68a614a3-3987-4614-a716-d188df5b383c"). InnerVolumeSpecName "kube-api-access-75z57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:03.541144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.541108 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75z57\" (UniqueName: \"kubernetes.io/projected/68a614a3-3987-4614-a716-d188df5b383c-kube-api-access-75z57\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.541144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.541139 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a614a3-3987-4614-a716-d188df5b383c-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:03.621924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.621890 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" path="/var/lib/kubelet/pods/c337afa1-a2b2-47fc-accb-4e11c7251b46/volumes" Nov 02 20:33:03.648623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.648575 5800 generic.go:356] "Generic (PLEG): container finished" podID="d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" containerID="c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809" exitCode=0 Nov 02 20:33:03.648854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.648833 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" Nov 02 20:33:03.648929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.648830 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" event={"ID":"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e","Type":"ContainerDied","Data":"c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809"} Nov 02 20:33:03.649008 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.648990 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb" event={"ID":"d5b30d57-44d0-42a7-b69d-2fa8cc1a320e","Type":"ContainerDied","Data":"a01d006c9b049cf936ef691a60a465c729d6bc31cb64c85058c30930a4312a56"} Nov 02 20:33:03.649072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.649025 5800 scope.go:122] "RemoveContainer" containerID="c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809" Nov 02 20:33:03.651312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.651250 5800 generic.go:356] "Generic (PLEG): container finished" podID="68a614a3-3987-4614-a716-d188df5b383c" containerID="65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495" exitCode=0 Nov 02 20:33:03.651453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.651381 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" Nov 02 20:33:03.651453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.651425 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" event={"ID":"68a614a3-3987-4614-a716-d188df5b383c","Type":"ContainerDied","Data":"65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495"} Nov 02 20:33:03.651547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.651467 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz" event={"ID":"68a614a3-3987-4614-a716-d188df5b383c","Type":"ContainerDied","Data":"6c36c8865231cd8c1330f26ec5f7a3e60ce6b23a647f7c26ce25c86b871ce12b"} Nov 02 20:33:03.661688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.661658 5800 scope.go:122] "RemoveContainer" containerID="c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809" Nov 02 20:33:03.662057 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:03.662033 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809\": container with ID starting with c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809 not found: ID does not exist" containerID="c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809" Nov 02 20:33:03.662155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.662065 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809"} err="failed to get container status \"c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809\": rpc error: code = NotFound desc = could not find container \"c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809\": container with ID starting with c4a9332f312f96a20de3947694d61593680625bbb6b48a59575450b78014a809 not found: ID does not exist" Nov 02 20:33:03.662155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.662092 5800 scope.go:122] "RemoveContainer" containerID="65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495" Nov 02 20:33:03.672743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.672721 5800 scope.go:122] "RemoveContainer" containerID="65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495" Nov 02 20:33:03.673079 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:03.673053 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495\": container with ID starting with 65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495 not found: ID does not exist" containerID="65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495" Nov 02 20:33:03.673160 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.673088 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495"} err="failed to get container status \"65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495\": rpc error: code = NotFound desc = could not find container \"65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495\": container with ID starting with 65e012ccafe57fd294a07d473ce929e54a0364769cefedd66b6070b835daf495 not found: ID does not exist" Nov 02 20:33:03.803922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.803885 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb"] Nov 02 20:33:03.808190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.808158 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-93816f-7d884fcf8f-6dzqb"] Nov 02 20:33:03.950095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.950054 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz"] Nov 02 20:33:03.953050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:03.953022 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-93816f-f79ff5db7-929xz"] Nov 02 20:33:04.550715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.550679 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5"] Nov 02 20:33:04.551266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551250 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" Nov 02 20:33:04.551266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551266 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551302 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" containerName="payload-processing" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551308 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" containerName="payload-processing" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551319 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551324 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551337 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d41a78d3-5375-4d5e-9746-bd27eefc0c27" containerName="revoke-keys" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551342 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41a78d3-5375-4d5e-9746-bd27eefc0c27" containerName="revoke-keys" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551360 5800 cpu_manager.go:414] "containerMap: removing container" podUID="68a614a3-3987-4614-a716-d188df5b383c" containerName="payload-pre-processing" Nov 02 20:33:04.551430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551366 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a614a3-3987-4614-a716-d188df5b383c" containerName="payload-pre-processing" Nov 02 20:33:04.551812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551464 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" Nov 02 20:33:04.551812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551475 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" containerName="payload-processing" Nov 02 20:33:04.551812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551481 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c337afa1-a2b2-47fc-accb-4e11c7251b46" containerName="istio-proxy" Nov 02 20:33:04.551812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551490 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41a78d3-5375-4d5e-9746-bd27eefc0c27" containerName="revoke-keys" Nov 02 20:33:04.551812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.551496 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a614a3-3987-4614-a716-d188df5b383c" containerName="payload-pre-processing" Nov 02 20:33:04.556324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.556293 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.608560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.608519 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5"] Nov 02 20:33:04.650121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.650092 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:33:04.650311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.650135 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:33:04.651456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.651433 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:04.651544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.651488 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:33:04.652846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.652809 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjskt\" (UniqueName: \"kubernetes.io/projected/1825e01e-96de-48a4-a634-e30bfe274475-kube-api-access-zjskt\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.652972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.652886 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.652972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.652944 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.753857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.753825 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.754023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.753879 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.754111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.754028 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjskt\" (UniqueName: \"kubernetes.io/projected/1825e01e-96de-48a4-a634-e30bfe274475-kube-api-access-zjskt\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.754739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.754709 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.756365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.756337 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.807847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.807762 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjskt\" (UniqueName: \"kubernetes.io/projected/1825e01e-96de-48a4-a634-e30bfe274475-kube-api-access-zjskt\") pod \"maas-api-revoke-sub-e81d3703-69kx5\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:04.871649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:04.871611 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:05.239976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:05.239942 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5"] Nov 02 20:33:05.240465 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:05.240439 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1825e01e_96de_48a4_a634_e30bfe274475.slice/crio-88ec0d66588fdde64a727244f76c1ed6c3f2bb4e0b767480e68dd58bea808aaf WatchSource:0}: Error finding container 88ec0d66588fdde64a727244f76c1ed6c3f2bb4e0b767480e68dd58bea808aaf: Status 404 returned error can't find the container with id 88ec0d66588fdde64a727244f76c1ed6c3f2bb4e0b767480e68dd58bea808aaf Nov 02 20:33:05.621867 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:05.621835 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a614a3-3987-4614-a716-d188df5b383c" path="/var/lib/kubelet/pods/68a614a3-3987-4614-a716-d188df5b383c/volumes" Nov 02 20:33:05.622241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:05.622227 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b30d57-44d0-42a7-b69d-2fa8cc1a320e" path="/var/lib/kubelet/pods/d5b30d57-44d0-42a7-b69d-2fa8cc1a320e/volumes" Nov 02 20:33:05.673583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:05.673543 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" event={"ID":"1825e01e-96de-48a4-a634-e30bfe274475","Type":"ContainerStarted","Data":"bfde71d4cf6cfb0aaf4c33a2c4aba2683c307e54d6ae0c6f4b1acd32ede4c014"} Nov 02 20:33:05.673763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:05.673604 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" event={"ID":"1825e01e-96de-48a4-a634-e30bfe274475","Type":"ContainerStarted","Data":"88ec0d66588fdde64a727244f76c1ed6c3f2bb4e0b767480e68dd58bea808aaf"} Nov 02 20:33:05.781009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:05.780952 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" podStartSLOduration=1.780938283 podStartE2EDuration="1.780938283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:05.778406832 +0000 UTC m=+2684.295179417" watchObservedRunningTime="2026-11-02 20:33:05.780938283 +0000 UTC m=+2684.297710868" Nov 02 20:33:06.682359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:06.682322 5800 generic.go:356] "Generic (PLEG): container finished" podID="1825e01e-96de-48a4-a634-e30bfe274475" containerID="bfde71d4cf6cfb0aaf4c33a2c4aba2683c307e54d6ae0c6f4b1acd32ede4c014" exitCode=0 Nov 02 20:33:06.682779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:06.682407 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" event={"ID":"1825e01e-96de-48a4-a634-e30bfe274475","Type":"ContainerDied","Data":"bfde71d4cf6cfb0aaf4c33a2c4aba2683c307e54d6ae0c6f4b1acd32ede4c014"} Nov 02 20:33:07.842503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:07.842471 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:07.988169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:07.988081 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-openshift-ca\") pod \"1825e01e-96de-48a4-a634-e30bfe274475\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " Nov 02 20:33:07.988392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:07.988211 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-xks-ca\") pod \"1825e01e-96de-48a4-a634-e30bfe274475\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " Nov 02 20:33:07.988392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:07.988276 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1825e01e-96de-48a4-a634-e30bfe274475-kube-api-access-zjskt\" (UniqueName: \"kubernetes.io/projected/1825e01e-96de-48a4-a634-e30bfe274475-kube-api-access-zjskt\") pod \"1825e01e-96de-48a4-a634-e30bfe274475\" (UID: \"1825e01e-96de-48a4-a634-e30bfe274475\") " Nov 02 20:33:07.989362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:07.989324 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-openshift-ca" pod "1825e01e-96de-48a4-a634-e30bfe274475" (UID: "1825e01e-96de-48a4-a634-e30bfe274475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:08.005883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.005613 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1825e01e-96de-48a4-a634-e30bfe274475-kube-api-access-zjskt" pod "1825e01e-96de-48a4-a634-e30bfe274475" (UID: "1825e01e-96de-48a4-a634-e30bfe274475"). InnerVolumeSpecName "kube-api-access-zjskt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:08.005883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.005748 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-xks-ca" pod "1825e01e-96de-48a4-a634-e30bfe274475" (UID: "1825e01e-96de-48a4-a634-e30bfe274475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:08.089749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.089711 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:08.089749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.089741 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e01e-96de-48a4-a634-e30bfe274475-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:08.089749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.089752 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjskt\" (UniqueName: \"kubernetes.io/projected/1825e01e-96de-48a4-a634-e30bfe274475-kube-api-access-zjskt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:08.699482 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.699449 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" Nov 02 20:33:08.699673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.699487 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5" event={"ID":"1825e01e-96de-48a4-a634-e30bfe274475","Type":"ContainerDied","Data":"88ec0d66588fdde64a727244f76c1ed6c3f2bb4e0b767480e68dd58bea808aaf"} Nov 02 20:33:08.699673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.699541 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ec0d66588fdde64a727244f76c1ed6c3f2bb4e0b767480e68dd58bea808aaf" Nov 02 20:33:08.950441 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.950361 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5"] Nov 02 20:33:08.950874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.950859 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1825e01e-96de-48a4-a634-e30bfe274475" containerName="revoke-keys" Nov 02 20:33:08.950922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.950875 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1825e01e-96de-48a4-a634-e30bfe274475" containerName="revoke-keys" Nov 02 20:33:08.951003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.950994 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1825e01e-96de-48a4-a634-e30bfe274475" containerName="revoke-keys" Nov 02 20:33:08.954693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:08.954667 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.003965 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.003929 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:09.004168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.004000 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:33:09.004168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.004087 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:33:09.007424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.007395 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5"] Nov 02 20:33:09.049250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.049222 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:33:09.100871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.100836 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.101054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.100900 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.101054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.101015 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8lr\" (UniqueName: \"kubernetes.io/projected/849d08ca-af7f-4c77-99a2-50cf60e06854-kube-api-access-2c8lr\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.202273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.202185 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.202273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.202242 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8lr\" (UniqueName: \"kubernetes.io/projected/849d08ca-af7f-4c77-99a2-50cf60e06854-kube-api-access-2c8lr\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.202503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.202341 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.203010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.202985 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.204897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.204875 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.255065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.255034 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8lr\" (UniqueName: \"kubernetes.io/projected/849d08ca-af7f-4c77-99a2-50cf60e06854-kube-api-access-2c8lr\") pod \"maas-api-revoke-sub-1307b358-sljq5\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.271039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.271013 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:09.637542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.637511 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5"] Nov 02 20:33:09.637704 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:09.637566 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849d08ca_af7f_4c77_99a2_50cf60e06854.slice/crio-b9fa2ce9bea8b01a4242060a5c0965c2f0e115a97b37fa091288f1314bbe02ea WatchSource:0}: Error finding container b9fa2ce9bea8b01a4242060a5c0965c2f0e115a97b37fa091288f1314bbe02ea: Status 404 returned error can't find the container with id b9fa2ce9bea8b01a4242060a5c0965c2f0e115a97b37fa091288f1314bbe02ea Nov 02 20:33:09.708193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:09.708160 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" event={"ID":"849d08ca-af7f-4c77-99a2-50cf60e06854","Type":"ContainerStarted","Data":"b9fa2ce9bea8b01a4242060a5c0965c2f0e115a97b37fa091288f1314bbe02ea"} Nov 02 20:33:10.717065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:10.717028 5800 generic.go:356] "Generic (PLEG): container finished" podID="849d08ca-af7f-4c77-99a2-50cf60e06854" containerID="ae25bc5bdc5108628f54735b0396de77491a05a02e79d06d0360edf7ab255ac2" exitCode=0 Nov 02 20:33:10.717587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:10.717110 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" event={"ID":"849d08ca-af7f-4c77-99a2-50cf60e06854","Type":"ContainerDied","Data":"ae25bc5bdc5108628f54735b0396de77491a05a02e79d06d0360edf7ab255ac2"} Nov 02 20:33:11.883837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:11.883810 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:12.033192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.033156 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-openshift-ca\") pod \"849d08ca-af7f-4c77-99a2-50cf60e06854\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " Nov 02 20:33:12.033431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.033259 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-xks-ca\") pod \"849d08ca-af7f-4c77-99a2-50cf60e06854\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " Nov 02 20:33:12.033431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.033357 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849d08ca-af7f-4c77-99a2-50cf60e06854-kube-api-access-2c8lr\" (UniqueName: \"kubernetes.io/projected/849d08ca-af7f-4c77-99a2-50cf60e06854-kube-api-access-2c8lr\") pod \"849d08ca-af7f-4c77-99a2-50cf60e06854\" (UID: \"849d08ca-af7f-4c77-99a2-50cf60e06854\") " Nov 02 20:33:12.033663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.033627 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-openshift-ca" pod "849d08ca-af7f-4c77-99a2-50cf60e06854" (UID: "849d08ca-af7f-4c77-99a2-50cf60e06854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:12.033770 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.033736 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:12.035472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.035443 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-xks-ca" pod "849d08ca-af7f-4c77-99a2-50cf60e06854" (UID: "849d08ca-af7f-4c77-99a2-50cf60e06854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:12.035652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.035628 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849d08ca-af7f-4c77-99a2-50cf60e06854-kube-api-access-2c8lr" pod "849d08ca-af7f-4c77-99a2-50cf60e06854" (UID: "849d08ca-af7f-4c77-99a2-50cf60e06854"). InnerVolumeSpecName "kube-api-access-2c8lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:12.135472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.135431 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8lr\" (UniqueName: \"kubernetes.io/projected/849d08ca-af7f-4c77-99a2-50cf60e06854-kube-api-access-2c8lr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:12.135472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.135472 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/849d08ca-af7f-4c77-99a2-50cf60e06854-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:12.737394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.737362 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" Nov 02 20:33:12.737605 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.737361 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5" event={"ID":"849d08ca-af7f-4c77-99a2-50cf60e06854","Type":"ContainerDied","Data":"b9fa2ce9bea8b01a4242060a5c0965c2f0e115a97b37fa091288f1314bbe02ea"} Nov 02 20:33:12.737605 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:12.737509 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9fa2ce9bea8b01a4242060a5c0965c2f0e115a97b37fa091288f1314bbe02ea" Nov 02 20:33:15.417485 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.417439 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz"] Nov 02 20:33:15.418260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.418235 5800 cpu_manager.go:414] "containerMap: removing container" podUID="849d08ca-af7f-4c77-99a2-50cf60e06854" containerName="revoke-keys" Nov 02 20:33:15.418414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.418261 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849d08ca-af7f-4c77-99a2-50cf60e06854" containerName="revoke-keys" Nov 02 20:33:15.418490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.418471 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="849d08ca-af7f-4c77-99a2-50cf60e06854" containerName="revoke-keys" Nov 02 20:33:15.422831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.422809 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.472252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.472140 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz"] Nov 02 20:33:15.522666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.522630 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:15.522882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.522727 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:33:15.522882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.522828 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:33:15.523032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.522988 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:33:15.574273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.574239 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.574501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.574311 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcgm9\" (UniqueName: \"kubernetes.io/projected/7498b75c-7199-4976-90ee-9a234d93a8fb-kube-api-access-gcgm9\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.574501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.574474 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.684390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.684236 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.684390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.684350 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcgm9\" (UniqueName: \"kubernetes.io/projected/7498b75c-7199-4976-90ee-9a234d93a8fb-kube-api-access-gcgm9\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.684706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.684468 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.687050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.687023 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.691536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.691503 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.738985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.738956 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcgm9\" (UniqueName: \"kubernetes.io/projected/7498b75c-7199-4976-90ee-9a234d93a8fb-kube-api-access-gcgm9\") pod \"maas-api-revoke-sub-246579f3-nzlnz\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:15.741144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:15.741122 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:16.044160 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:16.044127 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc"] Nov 02 20:33:16.047081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:16.047059 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e0174a-4v4xc"] Nov 02 20:33:16.145646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:16.145604 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz"] Nov 02 20:33:16.146386 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:16.146354 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7498b75c_7199_4976_90ee_9a234d93a8fb.slice/crio-9048f6b464a944bf1b5e597f1d13b7d520403293524ff547d7b21e9928684122 WatchSource:0}: Error finding container 9048f6b464a944bf1b5e597f1d13b7d520403293524ff547d7b21e9928684122: Status 404 returned error can't find the container with id 9048f6b464a944bf1b5e597f1d13b7d520403293524ff547d7b21e9928684122 Nov 02 20:33:16.771012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:16.770974 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" event={"ID":"7498b75c-7199-4976-90ee-9a234d93a8fb","Type":"ContainerStarted","Data":"59fd5573f073b5312efcf7e98f9e67bcf43f1025c985e16958f78b6c182383ee"} Nov 02 20:33:16.771520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:16.771030 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" event={"ID":"7498b75c-7199-4976-90ee-9a234d93a8fb","Type":"ContainerStarted","Data":"9048f6b464a944bf1b5e597f1d13b7d520403293524ff547d7b21e9928684122"} Nov 02 20:33:16.878918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:16.878856 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" podStartSLOduration=1.878836122 podStartE2EDuration="1.878836122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:16.877653297 +0000 UTC m=+2695.394425882" watchObservedRunningTime="2026-11-02 20:33:16.878836122 +0000 UTC m=+2695.395608708" Nov 02 20:33:17.621473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:17.621440 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f07eaaa-53cc-4410-bd6e-b8431f435d10" path="/var/lib/kubelet/pods/2f07eaaa-53cc-4410-bd6e-b8431f435d10/volumes" Nov 02 20:33:17.779108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:17.779074 5800 generic.go:356] "Generic (PLEG): container finished" podID="7498b75c-7199-4976-90ee-9a234d93a8fb" containerID="59fd5573f073b5312efcf7e98f9e67bcf43f1025c985e16958f78b6c182383ee" exitCode=0 Nov 02 20:33:17.779629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:17.779166 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" event={"ID":"7498b75c-7199-4976-90ee-9a234d93a8fb","Type":"ContainerDied","Data":"59fd5573f073b5312efcf7e98f9e67bcf43f1025c985e16958f78b6c182383ee"} Nov 02 20:33:18.948746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:18.948715 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:19.124446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.124355 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-openshift-ca\") pod \"7498b75c-7199-4976-90ee-9a234d93a8fb\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " Nov 02 20:33:19.124446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.124407 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7498b75c-7199-4976-90ee-9a234d93a8fb-kube-api-access-gcgm9\" (UniqueName: \"kubernetes.io/projected/7498b75c-7199-4976-90ee-9a234d93a8fb-kube-api-access-gcgm9\") pod \"7498b75c-7199-4976-90ee-9a234d93a8fb\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " Nov 02 20:33:19.124646 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.124520 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-xks-ca\") pod \"7498b75c-7199-4976-90ee-9a234d93a8fb\" (UID: \"7498b75c-7199-4976-90ee-9a234d93a8fb\") " Nov 02 20:33:19.124809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.124786 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-openshift-ca" pod "7498b75c-7199-4976-90ee-9a234d93a8fb" (UID: "7498b75c-7199-4976-90ee-9a234d93a8fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:19.124979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.124960 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:19.126713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.126678 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7498b75c-7199-4976-90ee-9a234d93a8fb-kube-api-access-gcgm9" pod "7498b75c-7199-4976-90ee-9a234d93a8fb" (UID: "7498b75c-7199-4976-90ee-9a234d93a8fb"). InnerVolumeSpecName "kube-api-access-gcgm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:19.127939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.127912 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-xks-ca" pod "7498b75c-7199-4976-90ee-9a234d93a8fb" (UID: "7498b75c-7199-4976-90ee-9a234d93a8fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:19.226203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.226145 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcgm9\" (UniqueName: \"kubernetes.io/projected/7498b75c-7199-4976-90ee-9a234d93a8fb-kube-api-access-gcgm9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:19.226203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.226203 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7498b75c-7199-4976-90ee-9a234d93a8fb-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:19.759603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.759566 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz"] Nov 02 20:33:19.760122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.760109 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7498b75c-7199-4976-90ee-9a234d93a8fb" containerName="revoke-keys" Nov 02 20:33:19.760168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.760125 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7498b75c-7199-4976-90ee-9a234d93a8fb" containerName="revoke-keys" Nov 02 20:33:19.760225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.760216 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7498b75c-7199-4976-90ee-9a234d93a8fb" containerName="revoke-keys" Nov 02 20:33:19.763738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.763723 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:19.795716 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.795687 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" Nov 02 20:33:19.795892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.795717 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz" event={"ID":"7498b75c-7199-4976-90ee-9a234d93a8fb","Type":"ContainerDied","Data":"9048f6b464a944bf1b5e597f1d13b7d520403293524ff547d7b21e9928684122"} Nov 02 20:33:19.795892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.795772 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9048f6b464a944bf1b5e597f1d13b7d520403293524ff547d7b21e9928684122" Nov 02 20:33:19.816631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.816596 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz"] Nov 02 20:33:19.935463 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.935371 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k"] Nov 02 20:33:19.935687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.935655 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5k9\" (UniqueName: \"kubernetes.io/projected/14f4b91d-b18c-492b-9fcb-0096a1c08515-kube-api-access-qk5k9\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:19.935839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.935815 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:19.935962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.935919 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:19.939568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.939547 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:19.992507 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:19.992469 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k"] Nov 02 20:33:20.037176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.037141 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5k9\" (UniqueName: \"kubernetes.io/projected/14f4b91d-b18c-492b-9fcb-0096a1c08515-kube-api-access-qk5k9\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:20.037391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.037238 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:20.037488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.037406 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:20.038200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.038171 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:20.040370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.040344 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:20.090133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.090097 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5k9\" (UniqueName: \"kubernetes.io/projected/14f4b91d-b18c-492b-9fcb-0096a1c08515-kube-api-access-qk5k9\") pod \"maas-api-revoke-sub-6641197c-s67hz\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:20.138988 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.138945 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035eb817-2660-479b-a30f-50997f181af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.139168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.139084 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258g7\" (UniqueName: \"kubernetes.io/projected/035eb817-2660-479b-a30f-50997f181af6-kube-api-access-258g7\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.139248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.139220 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035eb817-2660-479b-a30f-50997f181af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.240450 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.240362 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035eb817-2660-479b-a30f-50997f181af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.240627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.240457 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258g7\" (UniqueName: \"kubernetes.io/projected/035eb817-2660-479b-a30f-50997f181af6-kube-api-access-258g7\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.240627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.240531 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035eb817-2660-479b-a30f-50997f181af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.241203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.241172 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035eb817-2660-479b-a30f-50997f181af6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.243311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.243271 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035eb817-2660-479b-a30f-50997f181af6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.293750 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.293720 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-258g7\" (UniqueName: \"kubernetes.io/projected/035eb817-2660-479b-a30f-50997f181af6-kube-api-access-258g7\") pod \"maas-api-revoke-sub-bd2a0397-5wj6k\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.377428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.377394 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:20.554048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.554003 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:20.753271 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:20.753236 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f4b91d_b18c_492b_9fcb_0096a1c08515.slice/crio-8e0c63d35ae43487c2d3dbb2abee4d970352ffe7c227065b685306456e0a0fa7 WatchSource:0}: Error finding container 8e0c63d35ae43487c2d3dbb2abee4d970352ffe7c227065b685306456e0a0fa7: Status 404 returned error can't find the container with id 8e0c63d35ae43487c2d3dbb2abee4d970352ffe7c227065b685306456e0a0fa7 Nov 02 20:33:20.753530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.753508 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz"] Nov 02 20:33:20.803598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.803565 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" event={"ID":"14f4b91d-b18c-492b-9fcb-0096a1c08515","Type":"ContainerStarted","Data":"8e0c63d35ae43487c2d3dbb2abee4d970352ffe7c227065b685306456e0a0fa7"} Nov 02 20:33:20.930982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:20.930068 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k"] Nov 02 20:33:21.812137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:21.812098 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" event={"ID":"14f4b91d-b18c-492b-9fcb-0096a1c08515","Type":"ContainerStarted","Data":"ff2ad897804a6ca9d659a7fa51a65be9f3eba1b2b00183f8e60bd32e0be2e5f8"} Nov 02 20:33:21.813841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:21.813811 5800 generic.go:356] "Generic (PLEG): container finished" podID="035eb817-2660-479b-a30f-50997f181af6" containerID="2c2cc9a9c17df94d91ffc825a10cc0467d39d3dd341cddbb041723b671ae7a88" exitCode=0 Nov 02 20:33:21.813944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:21.813880 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" event={"ID":"035eb817-2660-479b-a30f-50997f181af6","Type":"ContainerDied","Data":"2c2cc9a9c17df94d91ffc825a10cc0467d39d3dd341cddbb041723b671ae7a88"} Nov 02 20:33:21.813944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:21.813919 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" event={"ID":"035eb817-2660-479b-a30f-50997f181af6","Type":"ContainerStarted","Data":"46e7e0b5ac3720f32742b52b7d6c43a2b000d5e42b91ceef1445cd4d34450ec1"} Nov 02 20:33:21.920472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:21.920423 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" podStartSLOduration=2.920409018 podStartE2EDuration="2.920409018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:21.918246409 +0000 UTC m=+2700.435018995" watchObservedRunningTime="2026-11-02 20:33:21.920409018 +0000 UTC m=+2700.437181603" Nov 02 20:33:22.127671 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:22.127585 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:33:22.135874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:22.135838 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:33:22.697095 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:22.697061 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:33:22.822492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:22.822458 5800 generic.go:356] "Generic (PLEG): container finished" podID="14f4b91d-b18c-492b-9fcb-0096a1c08515" containerID="ff2ad897804a6ca9d659a7fa51a65be9f3eba1b2b00183f8e60bd32e0be2e5f8" exitCode=0 Nov 02 20:33:22.822912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:22.822547 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" event={"ID":"14f4b91d-b18c-492b-9fcb-0096a1c08515","Type":"ContainerDied","Data":"ff2ad897804a6ca9d659a7fa51a65be9f3eba1b2b00183f8e60bd32e0be2e5f8"} Nov 02 20:33:22.994521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:22.994495 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:23.076127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.076090 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035eb817-2660-479b-a30f-50997f181af6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035eb817-2660-479b-a30f-50997f181af6-maas-api-openshift-ca\") pod \"035eb817-2660-479b-a30f-50997f181af6\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " Nov 02 20:33:23.076373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.076167 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035eb817-2660-479b-a30f-50997f181af6-kube-api-access-258g7\" (UniqueName: \"kubernetes.io/projected/035eb817-2660-479b-a30f-50997f181af6-kube-api-access-258g7\") pod \"035eb817-2660-479b-a30f-50997f181af6\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " Nov 02 20:33:23.076373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.076234 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/035eb817-2660-479b-a30f-50997f181af6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035eb817-2660-479b-a30f-50997f181af6-maas-api-xks-ca\") pod \"035eb817-2660-479b-a30f-50997f181af6\" (UID: \"035eb817-2660-479b-a30f-50997f181af6\") " Nov 02 20:33:23.076529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.076486 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035eb817-2660-479b-a30f-50997f181af6-maas-api-openshift-ca" pod "035eb817-2660-479b-a30f-50997f181af6" (UID: "035eb817-2660-479b-a30f-50997f181af6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:23.076660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.076644 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/035eb817-2660-479b-a30f-50997f181af6-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:23.078396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.078371 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035eb817-2660-479b-a30f-50997f181af6-maas-api-xks-ca" pod "035eb817-2660-479b-a30f-50997f181af6" (UID: "035eb817-2660-479b-a30f-50997f181af6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:23.078500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.078396 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035eb817-2660-479b-a30f-50997f181af6-kube-api-access-258g7" pod "035eb817-2660-479b-a30f-50997f181af6" (UID: "035eb817-2660-479b-a30f-50997f181af6"). InnerVolumeSpecName "kube-api-access-258g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:23.178242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.177519 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/035eb817-2660-479b-a30f-50997f181af6-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:23.178242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.177553 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258g7\" (UniqueName: \"kubernetes.io/projected/035eb817-2660-479b-a30f-50997f181af6-kube-api-access-258g7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:23.832607 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.832576 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" Nov 02 20:33:23.833117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.832607 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k" event={"ID":"035eb817-2660-479b-a30f-50997f181af6","Type":"ContainerDied","Data":"46e7e0b5ac3720f32742b52b7d6c43a2b000d5e42b91ceef1445cd4d34450ec1"} Nov 02 20:33:23.833117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:23.832659 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e7e0b5ac3720f32742b52b7d6c43a2b000d5e42b91ceef1445cd4d34450ec1" Nov 02 20:33:24.009195 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.009161 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:24.089266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.089230 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f4b91d-b18c-492b-9fcb-0096a1c08515-kube-api-access-qk5k9\" (UniqueName: \"kubernetes.io/projected/14f4b91d-b18c-492b-9fcb-0096a1c08515-kube-api-access-qk5k9\") pod \"14f4b91d-b18c-492b-9fcb-0096a1c08515\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " Nov 02 20:33:24.089492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.089306 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-openshift-ca\") pod \"14f4b91d-b18c-492b-9fcb-0096a1c08515\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " Nov 02 20:33:24.089492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.089389 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-xks-ca\") pod \"14f4b91d-b18c-492b-9fcb-0096a1c08515\" (UID: \"14f4b91d-b18c-492b-9fcb-0096a1c08515\") " Nov 02 20:33:24.089744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.089719 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-openshift-ca" pod "14f4b91d-b18c-492b-9fcb-0096a1c08515" (UID: "14f4b91d-b18c-492b-9fcb-0096a1c08515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:24.091591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.091564 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-xks-ca" pod "14f4b91d-b18c-492b-9fcb-0096a1c08515" (UID: "14f4b91d-b18c-492b-9fcb-0096a1c08515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:24.091591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.091576 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f4b91d-b18c-492b-9fcb-0096a1c08515-kube-api-access-qk5k9" pod "14f4b91d-b18c-492b-9fcb-0096a1c08515" (UID: "14f4b91d-b18c-492b-9fcb-0096a1c08515"). InnerVolumeSpecName "kube-api-access-qk5k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:24.200449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.191144 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk5k9\" (UniqueName: \"kubernetes.io/projected/14f4b91d-b18c-492b-9fcb-0096a1c08515-kube-api-access-qk5k9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:24.200449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.191181 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:24.200449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.191197 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14f4b91d-b18c-492b-9fcb-0096a1c08515-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:24.843746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.843708 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" Nov 02 20:33:24.843746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.843726 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz" event={"ID":"14f4b91d-b18c-492b-9fcb-0096a1c08515","Type":"ContainerDied","Data":"8e0c63d35ae43487c2d3dbb2abee4d970352ffe7c227065b685306456e0a0fa7"} Nov 02 20:33:24.844406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:24.843792 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0c63d35ae43487c2d3dbb2abee4d970352ffe7c227065b685306456e0a0fa7" Nov 02 20:33:27.223558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.223522 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl"] Nov 02 20:33:27.224070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.224054 5800 cpu_manager.go:414] "containerMap: removing container" podUID="035eb817-2660-479b-a30f-50997f181af6" containerName="revoke-keys" Nov 02 20:33:27.224111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.224071 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035eb817-2660-479b-a30f-50997f181af6" containerName="revoke-keys" Nov 02 20:33:27.224111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.224088 5800 cpu_manager.go:414] "containerMap: removing container" podUID="14f4b91d-b18c-492b-9fcb-0096a1c08515" containerName="revoke-keys" Nov 02 20:33:27.224111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.224093 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f4b91d-b18c-492b-9fcb-0096a1c08515" containerName="revoke-keys" Nov 02 20:33:27.224233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.224203 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f4b91d-b18c-492b-9fcb-0096a1c08515" containerName="revoke-keys" Nov 02 20:33:27.224233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.224222 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="035eb817-2660-479b-a30f-50997f181af6" containerName="revoke-keys" Nov 02 20:33:27.227862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.227838 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.270698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.270663 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g"] Nov 02 20:33:27.274796 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.274775 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.276963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.276934 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-426941-openshift-default-dockercfg-mxlvf\"" Nov 02 20:33:27.319341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.319306 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl"] Nov 02 20:33:27.328300 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.328258 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.328690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.328605 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.329120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.329093 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.330478 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.330513 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824qx\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-kube-api-access-824qx\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.330541 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.330571 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331343 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.331416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331383 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.331416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331407 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.331798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331455 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331486 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.331798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331581 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331607 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.331798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331632 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.331798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331651 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.331798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331690 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.332575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.331730 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv59v\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-kube-api-access-lv59v\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.332669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.330132 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g"] Nov 02 20:33:27.433562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433523 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.433562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433569 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433595 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433613 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433644 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433679 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv59v\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-kube-api-access-lv59v\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433715 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433733 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433750 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433775 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433792 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824qx\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-kube-api-access-824qx\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433810 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.433835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433829 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.434505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433858 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.434505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433873 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.434505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433890 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.434505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433918 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.434505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.433939 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.434758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.434643 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.434758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.434660 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.435144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.435115 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.435598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.435574 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.437974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.437944 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.438257 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.438236 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.438927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.438728 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.439020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.438993 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.439117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.439086 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.439235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.439147 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.439632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.439607 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.439866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.439846 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.440334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.440297 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.440705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.440683 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.490250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.490170 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.490250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.490171 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.490544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.490448 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-824qx\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-kube-api-access-824qx\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.490888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.490861 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv59v\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-kube-api-access-lv59v\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.543109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.543066 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:27.591150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.591115 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:27.888035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.887996 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw"] Nov 02 20:33:27.888312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.888263 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" containerID="cri-o://f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664" gracePeriod=30 Nov 02 20:33:27.962830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.962789 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl"] Nov 02 20:33:27.963955 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:27.963926 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ff5090_175b_45c7_9800_aee8c7cd57f1.slice/crio-c2f727ed7f6eb96c97aeda2edf767817f8f940156ac69c5a4427149b09adc9c7 WatchSource:0}: Error finding container c2f727ed7f6eb96c97aeda2edf767817f8f940156ac69c5a4427149b09adc9c7: Status 404 returned error can't find the container with id c2f727ed7f6eb96c97aeda2edf767817f8f940156ac69c5a4427149b09adc9c7 Nov 02 20:33:27.966854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.966818 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:27.966964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.966887 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:27.966964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:27.966918 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:28.015354 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:28.015231 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d4bf32_6cbb_4d68_aee2_265b1c9eb238.slice/crio-ac8a1029f39678b7b9f7b75fd16dd80c28c0e6eae055da7f6459fa124a764eee WatchSource:0}: Error finding container ac8a1029f39678b7b9f7b75fd16dd80c28c0e6eae055da7f6459fa124a764eee: Status 404 returned error can't find the container with id ac8a1029f39678b7b9f7b75fd16dd80c28c0e6eae055da7f6459fa124a764eee Nov 02 20:33:28.018431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.018386 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:28.018695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.018622 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:28.018695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.018681 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:28.019478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.019453 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g"] Nov 02 20:33:28.891412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.891359 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" event={"ID":"b1ff5090-175b-45c7-9800-aee8c7cd57f1","Type":"ContainerStarted","Data":"c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58"} Nov 02 20:33:28.891927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.891441 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" event={"ID":"b1ff5090-175b-45c7-9800-aee8c7cd57f1","Type":"ContainerStarted","Data":"c2f727ed7f6eb96c97aeda2edf767817f8f940156ac69c5a4427149b09adc9c7"} Nov 02 20:33:28.895362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.895325 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" event={"ID":"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238","Type":"ContainerStarted","Data":"7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b"} Nov 02 20:33:28.895486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:28.895385 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" event={"ID":"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238","Type":"ContainerStarted","Data":"ac8a1029f39678b7b9f7b75fd16dd80c28c0e6eae055da7f6459fa124a764eee"} Nov 02 20:33:29.013226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.013151 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podStartSLOduration=2.013130858 podStartE2EDuration="2.013130858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:29.011530422 +0000 UTC m=+2707.528303007" watchObservedRunningTime="2026-11-02 20:33:29.013130858 +0000 UTC m=+2707.529903444" Nov 02 20:33:29.133308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.125989 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podStartSLOduration=2.125969512 podStartE2EDuration="2.125969512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:29.122739704 +0000 UTC m=+2707.639512289" watchObservedRunningTime="2026-11-02 20:33:29.125969512 +0000 UTC m=+2707.642742098" Nov 02 20:33:29.551317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.547332 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:29.555307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.553646 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 02 20:33:29.555307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.553724 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 02 20:33:29.592704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.592592 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:29.593049 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.593018 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 02 20:33:29.593121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:29.593094 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 02 20:33:30.546193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:30.545725 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 02 20:33:30.546193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:30.545790 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 02 20:33:30.591974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:30.591925 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 02 20:33:30.592167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:30.592003 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 02 20:33:31.095468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.095428 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm"] Nov 02 20:33:31.099686 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.099668 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.150305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.150224 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:31.191305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.168942 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm"] Nov 02 20:33:31.198314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.194646 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:33:31.198314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.194747 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:33:31.198314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.194802 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:33:31.201951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.201918 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.202064 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.201977 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.202122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.202078 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tlp\" (UniqueName: \"kubernetes.io/projected/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-kube-api-access-f4tlp\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.302696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.302651 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.302865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.302718 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.302865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.302771 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tlp\" (UniqueName: \"kubernetes.io/projected/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-kube-api-access-f4tlp\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.303344 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.303322 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.305347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.305327 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.384450 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.378027 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tlp\" (UniqueName: \"kubernetes.io/projected/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-kube-api-access-f4tlp\") pod \"maas-api-revoke-sub-344bb3a5-x7hlm\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.414434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.414400 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:31.560631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.553390 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 02 20:33:31.560631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.553452 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 02 20:33:31.600909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.600788 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 02 20:33:31.600909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.600850 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 02 20:33:31.882350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.876042 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm"] Nov 02 20:33:31.888844 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:31.887593 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e37ad8_6e52_4bb2_bc4f_84db70d6b38e.slice/crio-474986e30d6c1649ca63fc025a1b453b78b036176fded26ea16934089a1bc591 WatchSource:0}: Error finding container 474986e30d6c1649ca63fc025a1b453b78b036176fded26ea16934089a1bc591: Status 404 returned error can't find the container with id 474986e30d6c1649ca63fc025a1b453b78b036176fded26ea16934089a1bc591 Nov 02 20:33:31.921069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:31.921038 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" event={"ID":"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e","Type":"ContainerStarted","Data":"474986e30d6c1649ca63fc025a1b453b78b036176fded26ea16934089a1bc591"} Nov 02 20:33:32.546306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:32.545980 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 02 20:33:32.546306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:32.546045 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 02 20:33:32.592100 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:32.592059 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 02 20:33:32.592548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:32.592110 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 02 20:33:32.935688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:32.934707 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" event={"ID":"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e","Type":"ContainerStarted","Data":"b2ef480c885398d9653515d86fca95df050a15c60fea6c3d1c005408e0dda94b"} Nov 02 20:33:33.305606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.305564 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:33:33.389232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.389190 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba9d419-bf87-4c95-9932-1954286e1a08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba9d419-bf87-4c95-9932-1954286e1a08-istiod-ca-cert\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.389536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.389514 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba9d419-bf87-4c95-9932-1954286e1a08-istiod-ca-cert" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:33.389619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.389591 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-envoy\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.389619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.389612 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-certs\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.389926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.389906 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-kube-api-access-29vc5\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-kube-api-access-29vc5\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.390021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.389967 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-credential-socket\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.390021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.389997 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-socket\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.390147 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390109 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-podinfo\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.390203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390180 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-data\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.390203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390197 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-token\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-token\") pod \"2ba9d419-bf87-4c95-9932-1954286e1a08\" (UID: \"2ba9d419-bf87-4c95-9932-1954286e1a08\") " Nov 02 20:33:33.390534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390482 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-certs" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:33.390895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390556 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-data" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:33.390895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390682 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-credential-socket" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:33.390895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390692 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-socket" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:33.390895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390781 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.390895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390802 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba9d419-bf87-4c95-9932-1954286e1a08-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.390895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.390819 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.391874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.391849 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-envoy" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:33:33.393393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.393371 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-podinfo" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:33:33.393532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.393486 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-kube-api-access-29vc5" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "kube-api-access-29vc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:33.393828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.393796 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-token" pod "2ba9d419-bf87-4c95-9932-1954286e1a08" (UID: "2ba9d419-bf87-4c95-9932-1954286e1a08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:33.469310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.468778 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" podStartSLOduration=2.468754095 podStartE2EDuration="2.468754095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:33.068864595 +0000 UTC m=+2711.585637187" watchObservedRunningTime="2026-11-02 20:33:33.468754095 +0000 UTC m=+2711.985526681" Nov 02 20:33:33.492379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.492113 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.492379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.492157 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29vc5\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-kube-api-access-29vc5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.492379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.492175 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.492379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.492189 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba9d419-bf87-4c95-9932-1954286e1a08-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.492379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.492204 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.492379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.492216 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba9d419-bf87-4c95-9932-1954286e1a08-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:33.548307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.543744 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 02 20:33:33.548307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.543819 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 02 20:33:33.591678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.591646 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 02 20:33:33.591917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.591891 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 02 20:33:33.970106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.970051 5800 generic.go:356] "Generic (PLEG): container finished" podID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerID="f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664" exitCode=0 Nov 02 20:33:33.970985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.970954 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" event={"ID":"2ba9d419-bf87-4c95-9932-1954286e1a08","Type":"ContainerDied","Data":"f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664"} Nov 02 20:33:33.971180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.971163 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" event={"ID":"2ba9d419-bf87-4c95-9932-1954286e1a08","Type":"ContainerDied","Data":"54584b9abc34ca24c2b09a98ff6ee339de5f8f122f544dfa04a8340fd94ea508"} Nov 02 20:33:33.971329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.971313 5800 scope.go:122] "RemoveContainer" containerID="f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664" Nov 02 20:33:33.971637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.971618 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw" Nov 02 20:33:33.981341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.981216 5800 generic.go:356] "Generic (PLEG): container finished" podID="b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" containerID="b2ef480c885398d9653515d86fca95df050a15c60fea6c3d1c005408e0dda94b" exitCode=0 Nov 02 20:33:33.981461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.981350 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" event={"ID":"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e","Type":"ContainerDied","Data":"b2ef480c885398d9653515d86fca95df050a15c60fea6c3d1c005408e0dda94b"} Nov 02 20:33:33.996593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:33.996574 5800 scope.go:122] "RemoveContainer" containerID="b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0" Nov 02 20:33:34.007880 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.007856 5800 scope.go:122] "RemoveContainer" containerID="f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664" Nov 02 20:33:34.008186 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:34.008162 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664\": container with ID starting with f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664 not found: ID does not exist" containerID="f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664" Nov 02 20:33:34.008256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.008195 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664"} err="failed to get container status \"f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664\": rpc error: code = NotFound desc = could not find container \"f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664\": container with ID starting with f64c6bea53efb348310c8f676ff7d4d130af37f86039927739af932db6fbd664 not found: ID does not exist" Nov 02 20:33:34.008256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.008213 5800 scope.go:122] "RemoveContainer" containerID="b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0" Nov 02 20:33:34.008615 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:34.008475 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0\": container with ID starting with b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0 not found: ID does not exist" containerID="b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0" Nov 02 20:33:34.008615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.008503 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0"} err="failed to get container status \"b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0\": rpc error: code = NotFound desc = could not find container \"b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0\": container with ID starting with b322bbe37fc46de8ae66a8dc6fffad7ec3cbba9c6dfc7994c8a54cb0095305c0 not found: ID does not exist" Nov 02 20:33:34.169214 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.169172 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw"] Nov 02 20:33:34.180854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.180823 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-93816f-openshift-default-85f9f7b5b9-p46sw"] Nov 02 20:33:34.544095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.544057 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 02 20:33:34.544311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.544123 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 02 20:33:34.591524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.591485 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 02 20:33:34.591713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:34.591552 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 02 20:33:35.172120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.172081 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:35.274036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.273995 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t"] Nov 02 20:33:35.274850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.274818 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" containerName="revoke-keys" Nov 02 20:33:35.274850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.274849 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" containerName="revoke-keys" Nov 02 20:33:35.275072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.274914 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" Nov 02 20:33:35.275072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.274923 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" Nov 02 20:33:35.275072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.274954 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" Nov 02 20:33:35.275072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.274965 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" Nov 02 20:33:35.275381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.275105 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" Nov 02 20:33:35.275381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.275139 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" containerName="revoke-keys" Nov 02 20:33:35.279469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.279439 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.318095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.316060 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-kube-api-access-f4tlp\" (UniqueName: \"kubernetes.io/projected/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-kube-api-access-f4tlp\") pod \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " Nov 02 20:33:35.318095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.316149 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-openshift-ca\") pod \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " Nov 02 20:33:35.318095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.316407 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-xks-ca\") pod \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\" (UID: \"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e\") " Nov 02 20:33:35.326705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.321480 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-openshift-ca" pod "b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" (UID: "b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:35.333477 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.332743 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-xks-ca" pod "b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" (UID: "b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:35.333477 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.333161 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bbee67-openshift-default-dockercfg-lw7rw\"" Nov 02 20:33:35.336690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.334535 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-kube-api-access-f4tlp" pod "b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" (UID: "b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e"). InnerVolumeSpecName "kube-api-access-f4tlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:35.380617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.380105 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t"] Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418259 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418361 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv6fp\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-kube-api-access-lv6fp\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418404 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418458 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418526 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418590 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418645 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418688 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418747 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418812 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418831 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:35.419003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.418847 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4tlp\" (UniqueName: \"kubernetes.io/projected/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e-kube-api-access-f4tlp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:35.520510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520462 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520537 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520594 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv6fp\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-kube-api-access-lv6fp\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520634 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520725 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520690 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520757 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520820 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520877 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.520996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.520919 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.522186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.521456 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.534740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.523674 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.535548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.534959 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.535548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.535203 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.535548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.535355 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.535548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.535492 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.540771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.538126 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.578304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.578142 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.580346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.578822 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv6fp\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-kube-api-access-lv6fp\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.608242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.608200 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:35.636617 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.630474 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" path="/var/lib/kubelet/pods/2ba9d419-bf87-4c95-9932-1954286e1a08/volumes" Nov 02 20:33:35.784960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.779669 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:35.784960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.781174 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:35.784960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.782050 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:33:35.895718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.895670 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:35.896132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.896002 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:35.897053 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:35.897027 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:33:36.002006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:36.001975 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" Nov 02 20:33:36.002234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:36.002024 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm" event={"ID":"b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e","Type":"ContainerDied","Data":"474986e30d6c1649ca63fc025a1b453b78b036176fded26ea16934089a1bc591"} Nov 02 20:33:36.002234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:36.002088 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474986e30d6c1649ca63fc025a1b453b78b036176fded26ea16934089a1bc591" Nov 02 20:33:36.088533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:36.088502 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t"] Nov 02 20:33:36.088922 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:36.088898 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4994ee_a43f_4eb7_ae76_44ffdf28a9ed.slice/crio-fd8c2e269d8df362c0b1d68a296b4a6fe41fc3dd4e9313ff4492aaded4e74af2 WatchSource:0}: Error finding container fd8c2e269d8df362c0b1d68a296b4a6fe41fc3dd4e9313ff4492aaded4e74af2: Status 404 returned error can't find the container with id fd8c2e269d8df362c0b1d68a296b4a6fe41fc3dd4e9313ff4492aaded4e74af2 Nov 02 20:33:36.091629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:36.091595 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:36.091738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:36.091679 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:36.091738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:36.091726 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:33:37.010458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:37.010418 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" event={"ID":"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed","Type":"ContainerStarted","Data":"6e5d5a17331d7ac84ee430d84c05d4b0d7d1adcd6274679aea5fd816c9a39e5d"} Nov 02 20:33:37.010971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:37.010500 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" event={"ID":"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed","Type":"ContainerStarted","Data":"fd8c2e269d8df362c0b1d68a296b4a6fe41fc3dd4e9313ff4492aaded4e74af2"} Nov 02 20:33:37.121513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:37.121461 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podStartSLOduration=2.121446717 podStartE2EDuration="2.121446717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:37.118022143 +0000 UTC m=+2715.634794728" watchObservedRunningTime="2026-11-02 20:33:37.121446717 +0000 UTC m=+2715.638219302" Nov 02 20:33:37.609309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:37.609266 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:37.611365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:37.611339 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 02 20:33:37.611547 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:37.611395 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 02 20:33:38.609545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:38.609508 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 02 20:33:38.610062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:38.609563 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 02 20:33:39.609401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:39.609364 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 02 20:33:39.609580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:39.609417 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 02 20:33:40.608860 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:40.608822 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 02 20:33:40.609047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:40.608886 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 02 20:33:41.616959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:41.616914 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 02 20:33:41.617499 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:41.617010 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 02 20:33:42.610301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:42.609848 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 02 20:33:42.610301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:42.609924 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 02 20:33:43.272209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.272164 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4"] Nov 02 20:33:43.273178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.273153 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba9d419-bf87-4c95-9932-1954286e1a08" containerName="istio-proxy" Nov 02 20:33:43.277632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.277611 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.328168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.328127 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4"] Nov 02 20:33:43.351235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.351190 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.351475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.351450 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2nr\" (UniqueName: \"kubernetes.io/projected/fc52b400-a81d-48cd-9323-fe8348a9a054-kube-api-access-2k2nr\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.351563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.351507 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.368478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.368443 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:43.368722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.368472 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:33:43.368722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.368472 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:33:43.372560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.372540 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:33:43.452434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.452394 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2nr\" (UniqueName: \"kubernetes.io/projected/fc52b400-a81d-48cd-9323-fe8348a9a054-kube-api-access-2k2nr\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.452654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.452448 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.452654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.452516 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.453161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.453133 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.455072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.455048 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.507234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.507204 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2nr\" (UniqueName: \"kubernetes.io/projected/fc52b400-a81d-48cd-9323-fe8348a9a054-kube-api-access-2k2nr\") pod \"maas-api-revoke-sub-16413107-mmqd4\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.593701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.593599 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:43.716702 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.716659 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:43.718167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.718115 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:43.721360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.721331 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:33:43.976478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:43.976450 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4"] Nov 02 20:33:43.978217 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:43.978187 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc52b400_a81d_48cd_9323_fe8348a9a054.slice/crio-c405bcf51cf9ed04e5238d4db245d8809b581b73938281b79e94cb658e38624b WatchSource:0}: Error finding container c405bcf51cf9ed04e5238d4db245d8809b581b73938281b79e94cb658e38624b: Status 404 returned error can't find the container with id c405bcf51cf9ed04e5238d4db245d8809b581b73938281b79e94cb658e38624b Nov 02 20:33:44.088395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:44.088356 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" event={"ID":"fc52b400-a81d-48cd-9323-fe8348a9a054","Type":"ContainerStarted","Data":"c405bcf51cf9ed04e5238d4db245d8809b581b73938281b79e94cb658e38624b"} Nov 02 20:33:45.100042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:45.099949 5800 generic.go:356] "Generic (PLEG): container finished" podID="fc52b400-a81d-48cd-9323-fe8348a9a054" containerID="36d0c26a327811259b52cc7a68edff7846bcf2f744d5acb6c84e320a2c8e79a0" exitCode=0 Nov 02 20:33:45.104321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:45.101347 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" event={"ID":"fc52b400-a81d-48cd-9323-fe8348a9a054","Type":"ContainerDied","Data":"36d0c26a327811259b52cc7a68edff7846bcf2f744d5acb6c84e320a2c8e79a0"} Nov 02 20:33:46.280897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.280863 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:46.387893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.387789 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc52b400-a81d-48cd-9323-fe8348a9a054-kube-api-access-2k2nr\" (UniqueName: \"kubernetes.io/projected/fc52b400-a81d-48cd-9323-fe8348a9a054-kube-api-access-2k2nr\") pod \"fc52b400-a81d-48cd-9323-fe8348a9a054\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " Nov 02 20:33:46.388062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.387892 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-xks-ca\") pod \"fc52b400-a81d-48cd-9323-fe8348a9a054\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " Nov 02 20:33:46.388062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.387969 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-openshift-ca\") pod \"fc52b400-a81d-48cd-9323-fe8348a9a054\" (UID: \"fc52b400-a81d-48cd-9323-fe8348a9a054\") " Nov 02 20:33:46.388631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.388577 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-openshift-ca" pod "fc52b400-a81d-48cd-9323-fe8348a9a054" (UID: "fc52b400-a81d-48cd-9323-fe8348a9a054"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:46.390552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.390511 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc52b400-a81d-48cd-9323-fe8348a9a054-kube-api-access-2k2nr" pod "fc52b400-a81d-48cd-9323-fe8348a9a054" (UID: "fc52b400-a81d-48cd-9323-fe8348a9a054"). InnerVolumeSpecName "kube-api-access-2k2nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:46.390741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.390719 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-xks-ca" pod "fc52b400-a81d-48cd-9323-fe8348a9a054" (UID: "fc52b400-a81d-48cd-9323-fe8348a9a054"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:46.489109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.489073 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:46.489109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.489102 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2nr\" (UniqueName: \"kubernetes.io/projected/fc52b400-a81d-48cd-9323-fe8348a9a054-kube-api-access-2k2nr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:46.489109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:46.489114 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc52b400-a81d-48cd-9323-fe8348a9a054-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:47.116823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:47.116742 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" Nov 02 20:33:47.116823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:47.116763 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4" event={"ID":"fc52b400-a81d-48cd-9323-fe8348a9a054","Type":"ContainerDied","Data":"c405bcf51cf9ed04e5238d4db245d8809b581b73938281b79e94cb658e38624b"} Nov 02 20:33:47.116823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:47.116816 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c405bcf51cf9ed04e5238d4db245d8809b581b73938281b79e94cb658e38624b" Nov 02 20:33:52.744334 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:33:52.743420 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:33:52.941695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:52.941645 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv"] Nov 02 20:33:52.947311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:52.946797 5800 cpu_manager.go:414] "containerMap: removing container" podUID="fc52b400-a81d-48cd-9323-fe8348a9a054" containerName="revoke-keys" Nov 02 20:33:52.947311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:52.946826 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc52b400-a81d-48cd-9323-fe8348a9a054" containerName="revoke-keys" Nov 02 20:33:52.947311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:52.947055 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc52b400-a81d-48cd-9323-fe8348a9a054" containerName="revoke-keys" Nov 02 20:33:52.953905 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:52.952811 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:52.996182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:52.996092 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:52.999561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:52.996629 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqw4d\" (UniqueName: \"kubernetes.io/projected/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-kube-api-access-hqw4d\") pod \"payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:53.020512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.012589 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv"] Nov 02 20:33:53.323310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.318598 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:53.323310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.318700 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqw4d\" (UniqueName: \"kubernetes.io/projected/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-kube-api-access-hqw4d\") pod \"payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:53.379347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.355815 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2c6ce9\"" Nov 02 20:33:53.379347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.355939 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2c6ce9-dockercfg-pgbfr\"" Nov 02 20:33:53.379347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.360140 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:53.424350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.414187 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqw4d\" (UniqueName: \"kubernetes.io/projected/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-kube-api-access-hqw4d\") pod \"payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:53.598988 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:53.598892 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:54.053924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.053877 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss"] Nov 02 20:33:54.058415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.058390 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.112427 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.112388 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss"] Nov 02 20:33:54.152917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.152822 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:33:54.152917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.152879 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:33:54.153175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.152937 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:33:54.153724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.153701 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:33:54.177327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.177257 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.177527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.177409 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.177631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.177606 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbtf\" (UniqueName: \"kubernetes.io/projected/fa74b8d1-9afa-4e9d-994a-5eaea0181490-kube-api-access-llbtf\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.272155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.272120 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv"] Nov 02 20:33:54.288324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.278855 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbtf\" (UniqueName: \"kubernetes.io/projected/fa74b8d1-9afa-4e9d-994a-5eaea0181490-kube-api-access-llbtf\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.288324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.279009 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.288324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.279057 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.288324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.280567 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.288324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.283653 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.332919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.332892 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbtf\" (UniqueName: \"kubernetes.io/projected/fa74b8d1-9afa-4e9d-994a-5eaea0181490-kube-api-access-llbtf\") pod \"maas-api-revoke-sub-bbd0381f-s25ss\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.374493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.374457 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:54.718197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.718099 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" event={"ID":"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6","Type":"ContainerStarted","Data":"4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e"} Nov 02 20:33:54.718197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.718163 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" event={"ID":"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6","Type":"ContainerStarted","Data":"109ac18dac459dbac5ce64b817e17c27c79360d7b95b5b088870750aec80a839"} Nov 02 20:33:54.718197 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.718195 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:33:54.777051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.777020 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss"] Nov 02 20:33:54.777976 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:33:54.777950 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa74b8d1_9afa_4e9d_994a_5eaea0181490.slice/crio-b96921e1a0547dd81624e1ed6a8e0c0ef3e0de8fbe1cf3c088b02faafcf2d3cf WatchSource:0}: Error finding container b96921e1a0547dd81624e1ed6a8e0c0ef3e0de8fbe1cf3c088b02faafcf2d3cf: Status 404 returned error can't find the container with id b96921e1a0547dd81624e1ed6a8e0c0ef3e0de8fbe1cf3c088b02faafcf2d3cf Nov 02 20:33:54.827471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:54.827421 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" podStartSLOduration=2.827407292 podStartE2EDuration="2.827407292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:54.824147087 +0000 UTC m=+2733.340919673" watchObservedRunningTime="2026-11-02 20:33:54.827407292 +0000 UTC m=+2733.344179879" Nov 02 20:33:55.727400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:55.727354 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" event={"ID":"fa74b8d1-9afa-4e9d-994a-5eaea0181490","Type":"ContainerStarted","Data":"976e526439aaf7e80e70e0e5f225a46fa25911a112efccd66aac8348d06efd58"} Nov 02 20:33:55.727852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:55.727429 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" event={"ID":"fa74b8d1-9afa-4e9d-994a-5eaea0181490","Type":"ContainerStarted","Data":"b96921e1a0547dd81624e1ed6a8e0c0ef3e0de8fbe1cf3c088b02faafcf2d3cf"} Nov 02 20:33:56.739167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:56.739125 5800 generic.go:356] "Generic (PLEG): container finished" podID="fa74b8d1-9afa-4e9d-994a-5eaea0181490" containerID="976e526439aaf7e80e70e0e5f225a46fa25911a112efccd66aac8348d06efd58" exitCode=0 Nov 02 20:33:56.739722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:56.739199 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" event={"ID":"fa74b8d1-9afa-4e9d-994a-5eaea0181490","Type":"ContainerDied","Data":"976e526439aaf7e80e70e0e5f225a46fa25911a112efccd66aac8348d06efd58"} Nov 02 20:33:56.918658 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:56.918626 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:57.016103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.016068 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa74b8d1-9afa-4e9d-994a-5eaea0181490-kube-api-access-llbtf\" (UniqueName: \"kubernetes.io/projected/fa74b8d1-9afa-4e9d-994a-5eaea0181490-kube-api-access-llbtf\") pod \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " Nov 02 20:33:57.016303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.016142 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-openshift-ca\") pod \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " Nov 02 20:33:57.016303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.016187 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-xks-ca\") pod \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\" (UID: \"fa74b8d1-9afa-4e9d-994a-5eaea0181490\") " Nov 02 20:33:57.016544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.016520 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-openshift-ca" pod "fa74b8d1-9afa-4e9d-994a-5eaea0181490" (UID: "fa74b8d1-9afa-4e9d-994a-5eaea0181490"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:33:57.018318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.018291 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-xks-ca" pod "fa74b8d1-9afa-4e9d-994a-5eaea0181490" (UID: "fa74b8d1-9afa-4e9d-994a-5eaea0181490"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:33:57.018436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.018415 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa74b8d1-9afa-4e9d-994a-5eaea0181490-kube-api-access-llbtf" pod "fa74b8d1-9afa-4e9d-994a-5eaea0181490" (UID: "fa74b8d1-9afa-4e9d-994a-5eaea0181490"). InnerVolumeSpecName "kube-api-access-llbtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:33:57.117545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.117511 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:57.117545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.117542 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llbtf\" (UniqueName: \"kubernetes.io/projected/fa74b8d1-9afa-4e9d-994a-5eaea0181490-kube-api-access-llbtf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:57.117545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.117553 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa74b8d1-9afa-4e9d-994a-5eaea0181490-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:33:57.748478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.748443 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" Nov 02 20:33:57.748909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.748552 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss" event={"ID":"fa74b8d1-9afa-4e9d-994a-5eaea0181490","Type":"ContainerDied","Data":"b96921e1a0547dd81624e1ed6a8e0c0ef3e0de8fbe1cf3c088b02faafcf2d3cf"} Nov 02 20:33:57.748909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:33:57.748607 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b96921e1a0547dd81624e1ed6a8e0c0ef3e0de8fbe1cf3c088b02faafcf2d3cf" Nov 02 20:34:02.883272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.883223 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd"] Nov 02 20:34:02.883830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.883813 5800 cpu_manager.go:414] "containerMap: removing container" podUID="fa74b8d1-9afa-4e9d-994a-5eaea0181490" containerName="revoke-keys" Nov 02 20:34:02.883885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.883830 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa74b8d1-9afa-4e9d-994a-5eaea0181490" containerName="revoke-keys" Nov 02 20:34:02.884002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.883990 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa74b8d1-9afa-4e9d-994a-5eaea0181490" containerName="revoke-keys" Nov 02 20:34:02.888930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.888909 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:02.984679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.984640 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-44a484\"" Nov 02 20:34:02.985233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.985055 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-44a484-dockercfg-64tlr\"" Nov 02 20:34:03.003345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:02.999877 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4t6\" (UniqueName: \"kubernetes.io/projected/765773e3-7f3a-4716-b29f-074cfda0758a-kube-api-access-2l4t6\") pod \"payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.003779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.003743 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765773e3-7f3a-4716-b29f-074cfda0758a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.029202 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.029166 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd"] Nov 02 20:34:03.038273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.038244 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5"] Nov 02 20:34:03.042588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.042346 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.095624 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.095591 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5"] Nov 02 20:34:03.104694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.104655 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4t6\" (UniqueName: \"kubernetes.io/projected/765773e3-7f3a-4716-b29f-074cfda0758a-kube-api-access-2l4t6\") pod \"payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.104856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.104750 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765773e3-7f3a-4716-b29f-074cfda0758a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.104856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.104779 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc4kk\" (UniqueName: \"kubernetes.io/projected/292e32f3-d4ed-44e6-9573-94c458727f4d-kube-api-access-bc4kk\") pod \"payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.104856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.104829 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e32f3-d4ed-44e6-9573-94c458727f4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.106058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.106021 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765773e3-7f3a-4716-b29f-074cfda0758a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.158299 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.158208 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4t6\" (UniqueName: \"kubernetes.io/projected/765773e3-7f3a-4716-b29f-074cfda0758a-kube-api-access-2l4t6\") pod \"payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.202308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.202267 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.205818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.205439 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e32f3-d4ed-44e6-9573-94c458727f4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.205818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.205593 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc4kk\" (UniqueName: \"kubernetes.io/projected/292e32f3-d4ed-44e6-9573-94c458727f4d-kube-api-access-bc4kk\") pod \"payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.206231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.206208 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e32f3-d4ed-44e6-9573-94c458727f4d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.261847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.261814 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc4kk\" (UniqueName: \"kubernetes.io/projected/292e32f3-d4ed-44e6-9573-94c458727f4d-kube-api-access-bc4kk\") pod \"payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.357356 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.357323 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.574615 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:34:03.574574 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765773e3_7f3a_4716_b29f_074cfda0758a.slice/crio-a5d82ed55a831b9d10d2cd5e3eb0dca2e3e8486672d3c46890b3f72565e94853 WatchSource:0}: Error finding container a5d82ed55a831b9d10d2cd5e3eb0dca2e3e8486672d3c46890b3f72565e94853: Status 404 returned error can't find the container with id a5d82ed55a831b9d10d2cd5e3eb0dca2e3e8486672d3c46890b3f72565e94853 Nov 02 20:34:03.574898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.574876 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd"] Nov 02 20:34:03.726850 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:34:03.726810 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292e32f3_d4ed_44e6_9573_94c458727f4d.slice/crio-8ab0e437bf4ebcb8d0dc7b0776bcc52598efb2bc78f25914abea31879b20386a WatchSource:0}: Error finding container 8ab0e437bf4ebcb8d0dc7b0776bcc52598efb2bc78f25914abea31879b20386a: Status 404 returned error can't find the container with id 8ab0e437bf4ebcb8d0dc7b0776bcc52598efb2bc78f25914abea31879b20386a Nov 02 20:34:03.727007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.726846 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5"] Nov 02 20:34:03.794651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.794618 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" event={"ID":"765773e3-7f3a-4716-b29f-074cfda0758a","Type":"ContainerStarted","Data":"d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f"} Nov 02 20:34:03.794805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.794676 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" event={"ID":"765773e3-7f3a-4716-b29f-074cfda0758a","Type":"ContainerStarted","Data":"a5d82ed55a831b9d10d2cd5e3eb0dca2e3e8486672d3c46890b3f72565e94853"} Nov 02 20:34:03.794867 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.794839 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:03.796419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.796390 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" event={"ID":"292e32f3-d4ed-44e6-9573-94c458727f4d","Type":"ContainerStarted","Data":"8ab0e437bf4ebcb8d0dc7b0776bcc52598efb2bc78f25914abea31879b20386a"} Nov 02 20:34:03.796532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.796451 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:03.904293 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:03.904223 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" podStartSLOduration=1.904208223 podStartE2EDuration="1.904208223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:03.902160822 +0000 UTC m=+2742.418933408" watchObservedRunningTime="2026-11-02 20:34:03.904208223 +0000 UTC m=+2742.420980807" Nov 02 20:34:04.011521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:04.011465 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" podStartSLOduration=2.011450107 podStartE2EDuration="2.011450107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:04.007351042 +0000 UTC m=+2742.524123627" watchObservedRunningTime="2026-11-02 20:34:04.011450107 +0000 UTC m=+2742.528222692" Nov 02 20:34:04.805453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:04.805412 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" event={"ID":"292e32f3-d4ed-44e6-9573-94c458727f4d","Type":"ContainerStarted","Data":"20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9"} Nov 02 20:34:05.731151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:05.731118 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 20:34:12.252475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.252435 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2"] Nov 02 20:34:12.259184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.259162 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.347474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347431 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvd6w\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-kube-api-access-hvd6w\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.347681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347592 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.347763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347660 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.347763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347732 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.347876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347764 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049812e6-2949-4744-9381-3b3ab5290247-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.347876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347793 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.347876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347837 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.348051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347942 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.348051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.347973 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049812e6-2949-4744-9381-3b3ab5290247-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449356 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvd6w\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-kube-api-access-hvd6w\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449450 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449504 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449553 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449578 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049812e6-2949-4744-9381-3b3ab5290247-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449603 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449639 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449730 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.449805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.449755 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049812e6-2949-4744-9381-3b3ab5290247-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.450993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.450624 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-data\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.450993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.450673 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049812e6-2949-4744-9381-3b3ab5290247-istiod-ca-cert\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.450993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.450734 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.450993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.450954 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-certs\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.452223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.452196 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-credential-socket\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.452403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.452374 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2"] Nov 02 20:34:12.453046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.453021 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-envoy\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.456023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.455029 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049812e6-2949-4744-9381-3b3ab5290247-istio-podinfo\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.502889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.502811 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-istio-token\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.505752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.505723 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvd6w\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-kube-api-access-hvd6w\") pod \"e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:12.578776 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:12.578735 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:13.005143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:13.005109 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2"] Nov 02 20:34:13.005391 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:34:13.005368 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049812e6_2949_4744_9381_3b3ab5290247.slice/crio-3a96d3bbc1976f1cdf0d9394605bd964d4d311b54299ab1f319dfb2ef96c95ed WatchSource:0}: Error finding container 3a96d3bbc1976f1cdf0d9394605bd964d4d311b54299ab1f319dfb2ef96c95ed: Status 404 returned error can't find the container with id 3a96d3bbc1976f1cdf0d9394605bd964d4d311b54299ab1f319dfb2ef96c95ed Nov 02 20:34:13.007736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:13.007704 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:13.007814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:13.007774 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:13.007814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:13.007804 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:13.891976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:13.891915 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" event={"ID":"049812e6-2949-4744-9381-3b3ab5290247","Type":"ContainerStarted","Data":"d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d"} Nov 02 20:34:13.892771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:13.891992 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" event={"ID":"049812e6-2949-4744-9381-3b3ab5290247","Type":"ContainerStarted","Data":"3a96d3bbc1976f1cdf0d9394605bd964d4d311b54299ab1f319dfb2ef96c95ed"} Nov 02 20:34:14.092756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:14.092691 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podStartSLOduration=2.092672212 podStartE2EDuration="2.092672212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:14.050760336 +0000 UTC m=+2752.567532922" watchObservedRunningTime="2026-11-02 20:34:14.092672212 +0000 UTC m=+2752.609444797" Nov 02 20:34:14.579359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:14.579320 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:14.580901 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:14.580871 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 02 20:34:14.581043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:14.580921 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 02 20:34:14.809749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:14.809719 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 20:34:15.579942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:15.579905 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 02 20:34:15.580379 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:15.579964 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 02 20:34:15.845353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:15.842472 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 20:34:16.579246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:16.579210 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 02 20:34:16.579496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:16.579271 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 02 20:34:17.579724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:17.579680 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 02 20:34:17.580268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:17.579743 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 02 20:34:18.591306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:18.590667 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 02 20:34:18.591306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:18.590747 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 02 20:34:19.579396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:19.579359 5800 patch_prober.go:29] interesting pod/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body= Nov 02 20:34:19.579596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:19.579424 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" Nov 02 20:34:20.959903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:20.959857 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:20.960427 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:20.960193 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:20.961407 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:20.961385 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:34:22.698885 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:34:22.698849 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:34:23.661704 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.661668 5800 scope.go:122] "RemoveContainer" containerID="1b0e96cf8f5e2acc1b0db7f4d8cadf54af8cb7df6b0f2da58a84527652e0c84c" Nov 02 20:34:23.673111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.673077 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn"] Nov 02 20:34:23.679152 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.679131 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.736002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.731951 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn"] Nov 02 20:34:23.771326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.771275 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-76c687-openshift-default-dockercfg-2hl6v\"" Nov 02 20:34:23.862981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.862893 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-podinfo\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.862981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.862990 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-data\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.863268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.863070 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2b029d-51c6-4959-8c8a-a8484a34907a-istiod-ca-cert\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.863268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.863114 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt9gh\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-kube-api-access-xt9gh\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.863268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.863147 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-certs\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.863268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.863235 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-socket\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.863268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.863265 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-envoy\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.863587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.863334 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-credential-socket\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.863587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.863419 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-token\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.966513 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-credential-socket\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.966606 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-token\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.966662 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-podinfo\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.966695 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-data\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.966770 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2b029d-51c6-4959-8c8a-a8484a34907a-istiod-ca-cert\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.966885 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt9gh\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-kube-api-access-xt9gh\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.966924 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-certs\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.967007 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-socket\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.967039 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-envoy\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.969355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.967720 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-credential-socket\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.976314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.975347 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-data\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.976314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.976022 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2b029d-51c6-4959-8c8a-a8484a34907a-istiod-ca-cert\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.976314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.976301 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-certs\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.976653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.976526 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-socket\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.982220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.982116 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-envoy\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:23.985224 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:23.982968 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-podinfo\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:24.020596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:24.020562 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-token\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:24.027653 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:24.027609 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt9gh\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-kube-api-access-xt9gh\") pod \"e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:24.298650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:24.298614 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:24.722749 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:34:24.722721 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2b029d_51c6_4959_8c8a_a8484a34907a.slice/crio-3cefc4c22f0ba50345b1b2c9f0a15f4819cd5adb2b28579e76a942c4451ef598 WatchSource:0}: Error finding container 3cefc4c22f0ba50345b1b2c9f0a15f4819cd5adb2b28579e76a942c4451ef598: Status 404 returned error can't find the container with id 3cefc4c22f0ba50345b1b2c9f0a15f4819cd5adb2b28579e76a942c4451ef598 Nov 02 20:34:24.723233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:24.723211 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn"] Nov 02 20:34:24.725119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:24.725088 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:24.725215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:24.725154 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:24.725215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:24.725185 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:25.039068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:25.039024 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" event={"ID":"5c2b029d-51c6-4959-8c8a-a8484a34907a","Type":"ContainerStarted","Data":"f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6"} Nov 02 20:34:25.039518 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:25.039094 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" event={"ID":"5c2b029d-51c6-4959-8c8a-a8484a34907a","Type":"ContainerStarted","Data":"3cefc4c22f0ba50345b1b2c9f0a15f4819cd5adb2b28579e76a942c4451ef598"} Nov 02 20:34:25.300044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:25.299826 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:25.308453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:25.308426 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:25.419518 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:25.419458 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" podStartSLOduration=2.4194444649999998 podStartE2EDuration="2.419444465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:25.152518864 +0000 UTC m=+2763.669291450" watchObservedRunningTime="2026-11-02 20:34:25.419444465 +0000 UTC m=+2763.936217115" Nov 02 20:34:26.046386 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:26.046351 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:26.047516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:26.047492 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:34:31.462846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.462811 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7"] Nov 02 20:34:31.468367 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.468342 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:31.519478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.519444 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7"] Nov 02 20:34:31.570236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.570203 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-76c687-dockercfg-7jzvv\"" Nov 02 20:34:31.678978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.570594 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-76c687\"" Nov 02 20:34:31.679208 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.601035 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9sfp\" (UniqueName: \"kubernetes.io/projected/ef4ac609-e8d7-4d97-b0db-7963f40d7416-kube-api-access-s9sfp\") pod \"payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:31.680413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.680385 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4ac609-e8d7-4d97-b0db-7963f40d7416-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:31.781868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.781829 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9sfp\" (UniqueName: \"kubernetes.io/projected/ef4ac609-e8d7-4d97-b0db-7963f40d7416-kube-api-access-s9sfp\") pod \"payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:31.782072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.781938 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4ac609-e8d7-4d97-b0db-7963f40d7416-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:31.782728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.782706 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4ac609-e8d7-4d97-b0db-7963f40d7416-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:31.835236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:31.835193 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9sfp\" (UniqueName: \"kubernetes.io/projected/ef4ac609-e8d7-4d97-b0db-7963f40d7416-kube-api-access-s9sfp\") pod \"payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:32.083730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:32.083632 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:32.461591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:32.461563 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7"] Nov 02 20:34:32.462531 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:34:32.462502 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4ac609_e8d7_4d97_b0db_7963f40d7416.slice/crio-685eb32857b08304226dd0bd22149127b61e6dbeb28dc644b0d3d01bf8762c86 WatchSource:0}: Error finding container 685eb32857b08304226dd0bd22149127b61e6dbeb28dc644b0d3d01bf8762c86: Status 404 returned error can't find the container with id 685eb32857b08304226dd0bd22149127b61e6dbeb28dc644b0d3d01bf8762c86 Nov 02 20:34:33.106020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:33.105976 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" event={"ID":"ef4ac609-e8d7-4d97-b0db-7963f40d7416","Type":"ContainerStarted","Data":"e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3"} Nov 02 20:34:33.106567 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:33.106063 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" event={"ID":"ef4ac609-e8d7-4d97-b0db-7963f40d7416","Type":"ContainerStarted","Data":"685eb32857b08304226dd0bd22149127b61e6dbeb28dc644b0d3d01bf8762c86"} Nov 02 20:34:33.106567 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:33.106220 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:33.216562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:33.216502 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" podStartSLOduration=2.216486477 podStartE2EDuration="2.216486477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:33.211847598 +0000 UTC m=+2771.728620183" watchObservedRunningTime="2026-11-02 20:34:33.216486477 +0000 UTC m=+2771.733259123" Nov 02 20:34:35.734847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.734815 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth"] Nov 02 20:34:35.740076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.740029 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.829154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829114 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.829487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829463 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.829669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829642 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.829828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829683 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.829828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829720 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.829828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829775 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.829828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829796 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6p4\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-kube-api-access-dp6p4\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.830069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829831 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.830069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.829865 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.841762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.841730 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth"] Nov 02 20:34:35.931182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931147 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931292 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931346 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931387 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931417 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931472 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931498 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6p4\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-kube-api-access-dp6p4\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931539 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931588 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931693 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.931939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.931788 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.932178 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.932152 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.932421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.932397 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.932504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.932431 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.933541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.933521 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.933940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.933923 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.984926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.984854 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:35.985065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:35.984952 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6p4\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-kube-api-access-dp6p4\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:36.055095 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:36.055052 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:36.476603 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:34:36.476560 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0ba614_2d5d_493c_9eb2_a6e648f657ab.slice/crio-04fcc0c2767731c5e175a635f49fb15a16406b90bc44de253514be9b97fd1e2c WatchSource:0}: Error finding container 04fcc0c2767731c5e175a635f49fb15a16406b90bc44de253514be9b97fd1e2c: Status 404 returned error can't find the container with id 04fcc0c2767731c5e175a635f49fb15a16406b90bc44de253514be9b97fd1e2c Nov 02 20:34:36.478391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:36.478370 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth"] Nov 02 20:34:36.479161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:36.479132 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:36.479231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:36.479192 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:36.479231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:36.479221 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:34:37.148491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:37.148446 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" event={"ID":"ab0ba614-2d5d-493c-9eb2-a6e648f657ab","Type":"ContainerStarted","Data":"232afaece1508610044454adec01fb7a895db07f0d13b19f5942e2df5e5ca347"} Nov 02 20:34:37.148983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:37.148517 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" event={"ID":"ab0ba614-2d5d-493c-9eb2-a6e648f657ab","Type":"ContainerStarted","Data":"04fcc0c2767731c5e175a635f49fb15a16406b90bc44de253514be9b97fd1e2c"} Nov 02 20:34:37.261676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:37.261604 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podStartSLOduration=2.261581007 podStartE2EDuration="2.261581007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:37.256004691 +0000 UTC m=+2775.772777277" watchObservedRunningTime="2026-11-02 20:34:37.261581007 +0000 UTC m=+2775.778353593" Nov 02 20:34:38.057397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:38.057341 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:38.058731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:38.058705 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 02 20:34:38.058834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:38.058764 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 02 20:34:39.057766 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:39.057729 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 02 20:34:39.058215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:39.057793 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 02 20:34:40.055993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.055942 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 02 20:34:40.056225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.056010 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 02 20:34:40.419263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.419181 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg"] Nov 02 20:34:40.423639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.423618 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:40.486209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.486169 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg"] Nov 02 20:34:40.501759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.501727 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-a9ed60-dockercfg-vvrcp\"" Nov 02 20:34:40.502090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.502070 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a9ed60\"" Nov 02 20:34:40.622003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.585914 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtnvl\" (UniqueName: \"kubernetes.io/projected/e7e14938-4991-4f94-ad15-9cf92a2e185d-kube-api-access-xtnvl\") pod \"payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:40.622003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.586029 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e14938-4991-4f94-ad15-9cf92a2e185d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:40.687069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.686969 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtnvl\" (UniqueName: \"kubernetes.io/projected/e7e14938-4991-4f94-ad15-9cf92a2e185d-kube-api-access-xtnvl\") pod \"payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:40.687069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.687067 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e14938-4991-4f94-ad15-9cf92a2e185d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:40.688054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.688025 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e14938-4991-4f94-ad15-9cf92a2e185d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:40.759682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.759649 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtnvl\" (UniqueName: \"kubernetes.io/projected/e7e14938-4991-4f94-ad15-9cf92a2e185d-kube-api-access-xtnvl\") pod \"payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:40.767582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:40.767556 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:41.056806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:41.056764 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 02 20:34:41.057012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:41.056832 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 02 20:34:41.164866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:41.164835 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg"] Nov 02 20:34:41.165184 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:34:41.165157 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e14938_4991_4f94_ad15_9cf92a2e185d.slice/crio-998498d32e9c7cc0d1b88702b145003a308ae9175f9ca7e6d8775d381d21b907 WatchSource:0}: Error finding container 998498d32e9c7cc0d1b88702b145003a308ae9175f9ca7e6d8775d381d21b907: Status 404 returned error can't find the container with id 998498d32e9c7cc0d1b88702b145003a308ae9175f9ca7e6d8775d381d21b907 Nov 02 20:34:41.278962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:41.278907 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" event={"ID":"e7e14938-4991-4f94-ad15-9cf92a2e185d","Type":"ContainerStarted","Data":"1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a"} Nov 02 20:34:41.279117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:41.278986 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" event={"ID":"e7e14938-4991-4f94-ad15-9cf92a2e185d","Type":"ContainerStarted","Data":"998498d32e9c7cc0d1b88702b145003a308ae9175f9ca7e6d8775d381d21b907"} Nov 02 20:34:41.280409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:41.280390 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:34:41.390695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:41.390583 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" podStartSLOduration=1.3905693559999999 podStartE2EDuration="1.390569356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:41.388411799 +0000 UTC m=+2779.905184413" watchObservedRunningTime="2026-11-02 20:34:41.390569356 +0000 UTC m=+2779.907341941" Nov 02 20:34:42.057662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:42.057626 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 02 20:34:42.058099 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:42.057685 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 02 20:34:43.055834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:43.055796 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Nov 02 20:34:43.056038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:43.055869 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Nov 02 20:34:44.118616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:44.118577 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 20:34:44.377789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:44.377673 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:44.395384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:44.395346 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:44.410585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:44.410546 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:34:52.696581 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:34:52.696546 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:34:53.300598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:34:53.300570 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 20:35:22.696892 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:35:22.696801 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:35:23.723982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:23.723943 5800 scope.go:122] "RemoveContainer" containerID="ee23638cab7c8ce15ddee4e359fbfc722e651f23d6e74b38653db79d3c773de3" Nov 02 20:35:37.055306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:37.054516 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g"] Nov 02 20:35:37.093476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:37.093428 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16d5e9ae-cn64g"] Nov 02 20:35:37.624528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:37.624497 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d5312a-39be-47fa-bd43-18372611fdf9" path="/var/lib/kubelet/pods/e2d5312a-39be-47fa-bd43-18372611fdf9/volumes" Nov 02 20:35:45.827450 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:45.827415 5800 generic.go:356] "Generic (PLEG): container finished" podID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerID="f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6" exitCode=0 Nov 02 20:35:45.828004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:45.827496 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" event={"ID":"5c2b029d-51c6-4959-8c8a-a8484a34907a","Type":"ContainerDied","Data":"f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6"} Nov 02 20:35:45.828072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:45.828002 5800 scope.go:122] "RemoveContainer" containerID="f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6" Nov 02 20:35:45.829822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:45.829783 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:45.829934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:45.829878 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:45.829934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:45.829927 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:46.858349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:46.858313 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" event={"ID":"5c2b029d-51c6-4959-8c8a-a8484a34907a","Type":"ContainerStarted","Data":"9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c"} Nov 02 20:35:47.299055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.299022 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:35:47.299343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.299321 5800 patch_prober.go:29] interesting pod/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 20:35:47.299430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.299384 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 20:35:47.399172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.399137 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh"] Nov 02 20:35:47.403488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.403468 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.462306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.458390 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-78bef7-openshift-default-dockercfg-n87nb\"" Nov 02 20:35:47.512641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.512585 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh"] Nov 02 20:35:47.696127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.696032 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22126ebf-024b-45aa-91d1-50488241b3ba-istio-podinfo\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.698237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.698196 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-certs\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.698589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.698562 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22126ebf-024b-45aa-91d1-50488241b3ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.698733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.698715 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-istio-token\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.699107 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.699087 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-data\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.699249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.699234 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-envoy\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.699414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.699396 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncq6z\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-kube-api-access-ncq6z\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.699545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.699531 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-socket\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.699729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.699715 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-credential-socket\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.801536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.801497 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22126ebf-024b-45aa-91d1-50488241b3ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802146 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802126 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22126ebf-024b-45aa-91d1-50488241b3ba-istiod-ca-cert\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802169 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-istio-token\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802227 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-data\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802247 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-envoy\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802367 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncq6z\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-kube-api-access-ncq6z\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802444 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802414 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-socket\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802479 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-credential-socket\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802537 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-data\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802583 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22126ebf-024b-45aa-91d1-50488241b3ba-istio-podinfo\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802680 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802613 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-certs\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802786 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-socket\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.802866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.802827 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-credential-socket\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.803030 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.803008 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-certs\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.804601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.804578 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-envoy\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.805094 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.805072 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22126ebf-024b-45aa-91d1-50488241b3ba-istio-podinfo\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.854969 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.854928 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncq6z\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-kube-api-access-ncq6z\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:47.855211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:47.855186 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-istio-token\") pod \"e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:48.017083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.017051 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:48.299790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.299707 5800 patch_prober.go:29] interesting pod/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 20:35:48.299790 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.299765 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 20:35:48.464756 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:35:48.464723 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22126ebf_024b_45aa_91d1_50488241b3ba.slice/crio-80bacf54d08f44d1cf2987ca13a43877fce54fa2dab30c980bb936d51145296a WatchSource:0}: Error finding container 80bacf54d08f44d1cf2987ca13a43877fce54fa2dab30c980bb936d51145296a: Status 404 returned error can't find the container with id 80bacf54d08f44d1cf2987ca13a43877fce54fa2dab30c980bb936d51145296a Nov 02 20:35:48.465229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.465203 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh"] Nov 02 20:35:48.467651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.467612 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:48.467771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.467694 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:48.467771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.467741 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:48.880931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.880832 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" event={"ID":"22126ebf-024b-45aa-91d1-50488241b3ba","Type":"ContainerStarted","Data":"cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e"} Nov 02 20:35:48.880931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.880902 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" event={"ID":"22126ebf-024b-45aa-91d1-50488241b3ba","Type":"ContainerStarted","Data":"80bacf54d08f44d1cf2987ca13a43877fce54fa2dab30c980bb936d51145296a"} Nov 02 20:35:48.991540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:48.991147 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podStartSLOduration=1.991133766 podStartE2EDuration="1.991133766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:48.988867072 +0000 UTC m=+2847.505639657" watchObservedRunningTime="2026-11-02 20:35:48.991133766 +0000 UTC m=+2847.507906350" Nov 02 20:35:49.017541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:49.017503 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:49.023293 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:49.023256 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:49.299911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:49.299873 5800 patch_prober.go:29] interesting pod/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 20:35:49.300111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:49.299929 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 20:35:49.889720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:49.889684 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:49.890624 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:49.890601 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:35:50.299062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:50.299023 5800 patch_prober.go:29] interesting pod/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 20:35:50.299601 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:50.299090 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 20:35:51.299156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:51.299117 5800 patch_prober.go:29] interesting pod/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 02 20:35:51.299603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:51.299174 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 02 20:35:52.304068 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:52.304028 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:35:52.304560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:52.304522 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:35:52.305383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:52.305358 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 20:35:52.700990 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:35:52.700907 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:35:55.236323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.236267 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q"] Nov 02 20:35:55.242721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.242695 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.289194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.289148 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q"] Nov 02 20:35:55.329663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.329056 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.329663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.329267 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p478x\" (UniqueName: \"kubernetes.io/projected/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-kube-api-access-p478x\") pod \"payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.335519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.335483 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-78bef7-dockercfg-t496f\"" Nov 02 20:35:55.335710 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.335553 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-78bef7\"" Nov 02 20:35:55.417666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.417609 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb"] Nov 02 20:35:55.435967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.435922 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.436486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.436465 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p478x\" (UniqueName: \"kubernetes.io/projected/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-kube-api-access-p478x\") pod \"payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.442960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.442917 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.449549 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.449523 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.463777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.463751 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb"] Nov 02 20:35:55.502916 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.502020 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p478x\" (UniqueName: \"kubernetes.io/projected/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-kube-api-access-p478x\") pod \"payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.542360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.542326 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rmp\" (UniqueName: \"kubernetes.io/projected/a0f5768d-4731-4a0f-9701-5f35fbb6129f-kube-api-access-n6rmp\") pod \"payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.542668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.542640 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f5768d-4731-4a0f-9701-5f35fbb6129f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.559665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.559636 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:55.643930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.643894 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f5768d-4731-4a0f-9701-5f35fbb6129f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.644075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.643982 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rmp\" (UniqueName: \"kubernetes.io/projected/a0f5768d-4731-4a0f-9701-5f35fbb6129f-kube-api-access-n6rmp\") pod \"payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.645414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.645388 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f5768d-4731-4a0f-9701-5f35fbb6129f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.699380 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.698512 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rmp\" (UniqueName: \"kubernetes.io/projected/a0f5768d-4731-4a0f-9701-5f35fbb6129f-kube-api-access-n6rmp\") pod \"payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.786841 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.786756 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:55.954367 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:35:55.954332 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bbf1c0_99cf_4945_88fa_34fdda2d2633.slice/crio-a4ec3c31f47506dba706b264012b50b03b34efd57331ec7ed93c151422910de7 WatchSource:0}: Error finding container a4ec3c31f47506dba706b264012b50b03b34efd57331ec7ed93c151422910de7: Status 404 returned error can't find the container with id a4ec3c31f47506dba706b264012b50b03b34efd57331ec7ed93c151422910de7 Nov 02 20:35:55.954750 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.954630 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q"] Nov 02 20:35:55.993413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:55.993358 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" event={"ID":"f7bbf1c0-99cf-4945-88fa-34fdda2d2633","Type":"ContainerStarted","Data":"a4ec3c31f47506dba706b264012b50b03b34efd57331ec7ed93c151422910de7"} Nov 02 20:35:56.051546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:56.051512 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm"] Nov 02 20:35:56.057306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:56.057259 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7d3cca-ntqzm"] Nov 02 20:35:56.164746 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:35:56.163837 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f5768d_4731_4a0f_9701_5f35fbb6129f.slice/crio-e207dcca425e10f5a368a12dc92ffe98abbdc25dd362e7f51e9b0f8007e573cd WatchSource:0}: Error finding container e207dcca425e10f5a368a12dc92ffe98abbdc25dd362e7f51e9b0f8007e573cd: Status 404 returned error can't find the container with id e207dcca425e10f5a368a12dc92ffe98abbdc25dd362e7f51e9b0f8007e573cd Nov 02 20:35:56.164746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:56.164410 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb"] Nov 02 20:35:57.002149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.002101 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" event={"ID":"f7bbf1c0-99cf-4945-88fa-34fdda2d2633","Type":"ContainerStarted","Data":"6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6"} Nov 02 20:35:57.002631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.002170 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:35:57.003858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.003835 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" event={"ID":"a0f5768d-4731-4a0f-9701-5f35fbb6129f","Type":"ContainerStarted","Data":"2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648"} Nov 02 20:35:57.003980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.003875 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" event={"ID":"a0f5768d-4731-4a0f-9701-5f35fbb6129f","Type":"ContainerStarted","Data":"e207dcca425e10f5a368a12dc92ffe98abbdc25dd362e7f51e9b0f8007e573cd"} Nov 02 20:35:57.003980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.003926 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:35:57.113217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.113162 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" podStartSLOduration=2.113146245 podStartE2EDuration="2.113146245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:57.109564154 +0000 UTC m=+2855.626336741" watchObservedRunningTime="2026-11-02 20:35:57.113146245 +0000 UTC m=+2855.629918830" Nov 02 20:35:57.214474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.214410 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" podStartSLOduration=2.214391939 podStartE2EDuration="2.214391939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:57.214364472 +0000 UTC m=+2855.731137059" watchObservedRunningTime="2026-11-02 20:35:57.214391939 +0000 UTC m=+2855.731164558" Nov 02 20:35:57.623008 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:35:57.622977 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6acab08b-2437-4f42-8c45-f706f4d4afec" path="/var/lib/kubelet/pods/6acab08b-2437-4f42-8c45-f706f4d4afec/volumes" Nov 02 20:36:08.014215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:08.014183 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 20:36:08.014754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:08.014267 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 20:36:11.943599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:11.943554 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj"] Nov 02 20:36:11.949224 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:11.949192 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.033307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.031042 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj"] Nov 02 20:36:12.046303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.042505 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-09ecd0-dockercfg-4n4z8\"" Nov 02 20:36:12.046303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.042601 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-09ecd0\"" Nov 02 20:36:12.102314 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.102195 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qt2f\" (UniqueName: \"kubernetes.io/projected/2559bedc-14d4-450e-8e85-8154603e1f57-kube-api-access-4qt2f\") pod \"payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.102830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.102786 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2559bedc-14d4-450e-8e85-8154603e1f57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.165124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.161865 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc"] Nov 02 20:36:12.170840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.170613 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94c63bd5-twhrc"] Nov 02 20:36:12.178466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.178438 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95"] Nov 02 20:36:12.194807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.194725 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95"] Nov 02 20:36:12.194977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.194870 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.203813 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.203783 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2559bedc-14d4-450e-8e85-8154603e1f57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.203981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.203858 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qt2f\" (UniqueName: \"kubernetes.io/projected/2559bedc-14d4-450e-8e85-8154603e1f57-kube-api-access-4qt2f\") pod \"payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.204881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.204858 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2559bedc-14d4-450e-8e85-8154603e1f57-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.256854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.256817 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qt2f\" (UniqueName: \"kubernetes.io/projected/2559bedc-14d4-450e-8e85-8154603e1f57-kube-api-access-4qt2f\") pod \"payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.264595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.264564 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:12.305055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.305003 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt574\" (UniqueName: \"kubernetes.io/projected/089f26f0-fc0d-4b7a-9238-b597f50aa558-kube-api-access-nt574\") pod \"payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.305332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.305220 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089f26f0-fc0d-4b7a-9238-b597f50aa558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.406690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.406642 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089f26f0-fc0d-4b7a-9238-b597f50aa558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.406912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.406752 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt574\" (UniqueName: \"kubernetes.io/projected/089f26f0-fc0d-4b7a-9238-b597f50aa558-kube-api-access-nt574\") pod \"payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.407581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.407557 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089f26f0-fc0d-4b7a-9238-b597f50aa558-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.460057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.459983 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt574\" (UniqueName: \"kubernetes.io/projected/089f26f0-fc0d-4b7a-9238-b597f50aa558-kube-api-access-nt574\") pod \"payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.510952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.510912 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:12.642966 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:36:12.642928 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2559bedc_14d4_450e_8e85_8154603e1f57.slice/crio-510cee138ab416fe833e5ccee3f9682771b63c8942e1402de58bd29bb69075e2 WatchSource:0}: Error finding container 510cee138ab416fe833e5ccee3f9682771b63c8942e1402de58bd29bb69075e2: Status 404 returned error can't find the container with id 510cee138ab416fe833e5ccee3f9682771b63c8942e1402de58bd29bb69075e2 Nov 02 20:36:12.643719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.643690 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj"] Nov 02 20:36:12.890273 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:36:12.890246 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089f26f0_fc0d_4b7a_9238_b597f50aa558.slice/crio-a5af78374b513a4400704c7c9d47cb24a0592b7d40cf5e64f28f3bdf4d4ab71c WatchSource:0}: Error finding container a5af78374b513a4400704c7c9d47cb24a0592b7d40cf5e64f28f3bdf4d4ab71c: Status 404 returned error can't find the container with id a5af78374b513a4400704c7c9d47cb24a0592b7d40cf5e64f28f3bdf4d4ab71c Nov 02 20:36:12.890420 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:12.890397 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95"] Nov 02 20:36:13.181127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.181086 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" event={"ID":"2559bedc-14d4-450e-8e85-8154603e1f57","Type":"ContainerStarted","Data":"0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051"} Nov 02 20:36:13.181650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.181155 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:13.181650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.181182 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" event={"ID":"2559bedc-14d4-450e-8e85-8154603e1f57","Type":"ContainerStarted","Data":"510cee138ab416fe833e5ccee3f9682771b63c8942e1402de58bd29bb69075e2"} Nov 02 20:36:13.183021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.182991 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" event={"ID":"089f26f0-fc0d-4b7a-9238-b597f50aa558","Type":"ContainerStarted","Data":"199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc"} Nov 02 20:36:13.183154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.183043 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" event={"ID":"089f26f0-fc0d-4b7a-9238-b597f50aa558","Type":"ContainerStarted","Data":"a5af78374b513a4400704c7c9d47cb24a0592b7d40cf5e64f28f3bdf4d4ab71c"} Nov 02 20:36:13.183154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.183069 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:13.290714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.290659 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" podStartSLOduration=2.290645382 podStartE2EDuration="2.290645382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:13.286424818 +0000 UTC m=+2871.803197402" watchObservedRunningTime="2026-11-02 20:36:13.290645382 +0000 UTC m=+2871.807417971" Nov 02 20:36:13.395618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.395534 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" podStartSLOduration=1.395517913 podStartE2EDuration="1.395517913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:13.392008792 +0000 UTC m=+2871.908781377" watchObservedRunningTime="2026-11-02 20:36:13.395517913 +0000 UTC m=+2871.912290497" Nov 02 20:36:13.626526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:13.626487 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4094d024-75b2-4c15-9f6d-e21e17fa36fe" path="/var/lib/kubelet/pods/4094d024-75b2-4c15-9f6d-e21e17fa36fe/volumes" Nov 02 20:36:22.695831 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:36:22.695791 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:36:24.194263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:24.194232 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 20:36:24.194712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:24.194471 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 20:36:28.068303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:28.068245 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf"] Nov 02 20:36:28.112806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:28.112772 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aeb0f1-m2ldf"] Nov 02 20:36:29.623162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:29.623127 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b57839f-6f4c-4595-865d-84c1a1f42ba4" path="/var/lib/kubelet/pods/4b57839f-6f4c-4595-865d-84c1a1f42ba4/volumes" Nov 02 20:36:41.348303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.336088 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp"] Nov 02 20:36:41.360315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.359357 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.408649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.408621 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-578636-openshift-default-dockercfg-gqmxj\"" Nov 02 20:36:41.441240 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.439494 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-credential-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.441640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.441617 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638407-68f5-4686-b8f4-3367e9009481-istiod-ca-cert\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.441851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.441832 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8lv2\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-kube-api-access-d8lv2\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.442090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.442063 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-data\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.442296 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.442264 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-certs\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.442482 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.442467 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638407-68f5-4686-b8f4-3367e9009481-istio-podinfo\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.442699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.442680 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-envoy\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.442891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.442876 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.443115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.443099 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-istio-token\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.517757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.517180 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp"] Nov 02 20:36:41.544844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.544805 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-istio-token\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.544961 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-credential-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545106 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638407-68f5-4686-b8f4-3367e9009481-istiod-ca-cert\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545137 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8lv2\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-kube-api-access-d8lv2\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545213 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-data\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545241 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-certs\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545293 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638407-68f5-4686-b8f4-3367e9009481-istio-podinfo\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545365 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-envoy\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545407 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545453 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-credential-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545624 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-data\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545857 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-certs\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545970 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545868 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-socket\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.545970 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.545900 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638407-68f5-4686-b8f4-3367e9009481-istiod-ca-cert\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.548054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.548028 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-envoy\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.548173 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.548155 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638407-68f5-4686-b8f4-3367e9009481-istio-podinfo\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.598244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.598210 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-istio-token\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.598472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.598342 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8lv2\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-kube-api-access-d8lv2\") pod \"e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:41.712037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:41.712008 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:42.056039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.056001 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl"] Nov 02 20:36:42.059791 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.059764 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591fa8e0-tx2bl"] Nov 02 20:36:42.183855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.183821 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp"] Nov 02 20:36:42.184079 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:36:42.184057 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32638407_68f5_4686_b8f4_3367e9009481.slice/crio-3e28867ccfd284fdfcae6407d629f94acfe86c97c79130e6d95735680baa079a WatchSource:0}: Error finding container 3e28867ccfd284fdfcae6407d629f94acfe86c97c79130e6d95735680baa079a: Status 404 returned error can't find the container with id 3e28867ccfd284fdfcae6407d629f94acfe86c97c79130e6d95735680baa079a Nov 02 20:36:42.187071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.187031 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:42.187187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.187146 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:42.187261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.187195 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:42.522151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.522111 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" event={"ID":"32638407-68f5-4686-b8f4-3367e9009481","Type":"ContainerStarted","Data":"d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1"} Nov 02 20:36:42.522644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.522175 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" event={"ID":"32638407-68f5-4686-b8f4-3367e9009481","Type":"ContainerStarted","Data":"3e28867ccfd284fdfcae6407d629f94acfe86c97c79130e6d95735680baa079a"} Nov 02 20:36:42.633339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.632269 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" podStartSLOduration=1.632252223 podStartE2EDuration="1.632252223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:42.629786073 +0000 UTC m=+2901.146558658" watchObservedRunningTime="2026-11-02 20:36:42.632252223 +0000 UTC m=+2901.149024807" Nov 02 20:36:42.712971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:42.712936 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:43.623732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:43.623697 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52dd166b-634f-456d-8488-89333dedb698" path="/var/lib/kubelet/pods/52dd166b-634f-456d-8488-89333dedb698/volumes" Nov 02 20:36:43.719357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:43.719324 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:44.538856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:44.538824 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:44.539864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:44.539843 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:36:45.044267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:45.044230 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8"] Nov 02 20:36:45.047852 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:45.047829 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b9a6631-tc6r8"] Nov 02 20:36:45.624545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:45.624514 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3722ae-b555-4a51-9784-0bb19aaa8c24" path="/var/lib/kubelet/pods/5b3722ae-b555-4a51-9784-0bb19aaa8c24/volumes" Nov 02 20:36:46.848306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:46.847848 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts"] Nov 02 20:36:46.855205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:46.855175 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:46.914310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:46.912610 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-6b12ae\"" Nov 02 20:36:46.934305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:46.931327 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31669d6-c91d-4a6b-9a10-282e9bb46d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:46.934305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:46.931382 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcjjm\" (UniqueName: \"kubernetes.io/projected/c31669d6-c91d-4a6b-9a10-282e9bb46d76-kube-api-access-zcjjm\") pod \"payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:46.968305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:46.967729 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-6b12ae-dockercfg-wvc7v\"" Nov 02 20:36:47.033518 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.033476 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31669d6-c91d-4a6b-9a10-282e9bb46d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:47.033787 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.033768 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcjjm\" (UniqueName: \"kubernetes.io/projected/c31669d6-c91d-4a6b-9a10-282e9bb46d76-kube-api-access-zcjjm\") pod \"payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:47.034573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.034446 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31669d6-c91d-4a6b-9a10-282e9bb46d76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:47.034573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.034505 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts"] Nov 02 20:36:47.089868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.089794 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcjjm\" (UniqueName: \"kubernetes.io/projected/c31669d6-c91d-4a6b-9a10-282e9bb46d76-kube-api-access-zcjjm\") pod \"payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:47.172589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.172546 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:47.552431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.552403 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts"] Nov 02 20:36:47.553496 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:36:47.553469 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31669d6_c91d_4a6b_9a10_282e9bb46d76.slice/crio-d1c865ac92117723e31f689ae11f7b013ec8f73165ca376015e251f0d45bd168 WatchSource:0}: Error finding container d1c865ac92117723e31f689ae11f7b013ec8f73165ca376015e251f0d45bd168: Status 404 returned error can't find the container with id d1c865ac92117723e31f689ae11f7b013ec8f73165ca376015e251f0d45bd168 Nov 02 20:36:47.571777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:47.571744 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" event={"ID":"c31669d6-c91d-4a6b-9a10-282e9bb46d76","Type":"ContainerStarted","Data":"d1c865ac92117723e31f689ae11f7b013ec8f73165ca376015e251f0d45bd168"} Nov 02 20:36:48.581764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:48.581727 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" event={"ID":"c31669d6-c91d-4a6b-9a10-282e9bb46d76","Type":"ContainerStarted","Data":"1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb"} Nov 02 20:36:48.582206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:48.581838 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:48.693579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:48.693522 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" podStartSLOduration=2.693506034 podStartE2EDuration="2.693506034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:48.689217148 +0000 UTC m=+2907.205989733" watchObservedRunningTime="2026-11-02 20:36:48.693506034 +0000 UTC m=+2907.210278618" Nov 02 20:36:51.564349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.563082 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79"] Nov 02 20:36:51.576453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.576423 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:51.609747 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.609697 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79"] Nov 02 20:36:51.637837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.637798 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkm62\" (UniqueName: \"kubernetes.io/projected/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-kube-api-access-dkm62\") pod \"payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:51.638037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.637882 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:51.649789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.649762 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-578636\"" Nov 02 20:36:51.650107 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.650071 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-578636-dockercfg-dj6lk\"" Nov 02 20:36:51.739139 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.739101 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkm62\" (UniqueName: \"kubernetes.io/projected/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-kube-api-access-dkm62\") pod \"payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:51.739376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.739176 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:51.740151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.740125 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:51.791959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.791929 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkm62\" (UniqueName: \"kubernetes.io/projected/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-kube-api-access-dkm62\") pod \"payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:51.909259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:51.909182 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:52.281273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:52.281246 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79"] Nov 02 20:36:52.281405 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:36:52.281340 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c11fc3a_992e_4371_8e21_523b5fc6b3b2.slice/crio-49b52c38797852fe2ffc0b070e6b3b01ace8216be5eb1d46448aa39bc8c0007d WatchSource:0}: Error finding container 49b52c38797852fe2ffc0b070e6b3b01ace8216be5eb1d46448aa39bc8c0007d: Status 404 returned error can't find the container with id 49b52c38797852fe2ffc0b070e6b3b01ace8216be5eb1d46448aa39bc8c0007d Nov 02 20:36:52.624519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:52.624426 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" event={"ID":"5c11fc3a-992e-4371-8e21-523b5fc6b3b2","Type":"ContainerStarted","Data":"5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f"} Nov 02 20:36:52.624519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:52.624488 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" event={"ID":"5c11fc3a-992e-4371-8e21-523b5fc6b3b2","Type":"ContainerStarted","Data":"49b52c38797852fe2ffc0b070e6b3b01ace8216be5eb1d46448aa39bc8c0007d"} Nov 02 20:36:52.624519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:52.624516 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:36:52.697745 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:36:52.697707 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:36:52.779227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:52.779170 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" podStartSLOduration=1.779156932 podStartE2EDuration="1.779156932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:52.729404327 +0000 UTC m=+2911.246176912" watchObservedRunningTime="2026-11-02 20:36:52.779156932 +0000 UTC m=+2911.295929511" Nov 02 20:36:56.045323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:56.045273 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp"] Nov 02 20:36:56.091433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:56.091398 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54cd9d82-lzmtp"] Nov 02 20:36:57.624233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:57.624197 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3986517e-1d56-42d0-9e97-35f6e18eab05" path="/var/lib/kubelet/pods/3986517e-1d56-42d0-9e97-35f6e18eab05/volumes" Nov 02 20:36:59.086296 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:59.086255 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56"] Nov 02 20:36:59.090146 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:59.090116 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac5bdd5e-z8h56"] Nov 02 20:36:59.592974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:59.592940 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 20:36:59.622980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:36:59.622942 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dfbe42-2a12-47c3-80d6-5616a232b401" path="/var/lib/kubelet/pods/36dfbe42-2a12-47c3-80d6-5616a232b401/volumes" Nov 02 20:37:03.639406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:03.639375 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 20:37:18.044871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:18.044829 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6"] Nov 02 20:37:18.090248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:18.090211 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8be8ab5-9j2q6"] Nov 02 20:37:19.680806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:19.680767 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5" path="/var/lib/kubelet/pods/f3cfe6c7-ef8e-4fae-b698-68bc4ce67aa5/volumes" Nov 02 20:37:20.097000 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:20.096949 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp"] Nov 02 20:37:20.103914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:20.103866 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07aca44-hw4wp"] Nov 02 20:37:20.816382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:20.816341 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs"] Nov 02 20:37:20.917587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:20.917554 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs"] Nov 02 20:37:20.917938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:20.917919 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.060305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.059825 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:37:21.060305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.059939 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:37:21.060305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.060041 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:37:21.060305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.060114 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:37:21.065303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.061776 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.065303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.061829 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.065303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.061866 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdbt\" (UniqueName: \"kubernetes.io/projected/c04c4357-a76e-46b2-9233-60395095c65c-kube-api-access-2jdbt\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.167415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.167310 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.167415 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.167391 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.167618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.167426 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdbt\" (UniqueName: \"kubernetes.io/projected/c04c4357-a76e-46b2-9233-60395095c65c-kube-api-access-2jdbt\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.168628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.168589 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.170552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.170522 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.220394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.220363 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdbt\" (UniqueName: \"kubernetes.io/projected/c04c4357-a76e-46b2-9233-60395095c65c-kube-api-access-2jdbt\") pod \"maas-api-revoke-sub-76b506af-znhjs\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.276358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.276315 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:21.630755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.629972 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbdea1c-7ae8-41e2-aec0-53d15f94b738" path="/var/lib/kubelet/pods/7dbdea1c-7ae8-41e2-aec0-53d15f94b738/volumes" Nov 02 20:37:21.703310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.700801 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs"] Nov 02 20:37:21.921999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:21.921906 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" event={"ID":"c04c4357-a76e-46b2-9233-60395095c65c","Type":"ContainerStarted","Data":"545ee6ef5cc2a01055cf97bf7e977a4a377e7ccf087445e66e8bc2bab91df55c"} Nov 02 20:37:22.804321 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:37:22.797937 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:37:22.951304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:22.949539 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" event={"ID":"c04c4357-a76e-46b2-9233-60395095c65c","Type":"ContainerStarted","Data":"8b24d3e937d3204735803554c296f4dbc7eb61b3d0820f9af33102b764267b16"} Nov 02 20:37:23.059662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:23.059549 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" podStartSLOduration=3.05953416 podStartE2EDuration="3.05953416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:23.056841629 +0000 UTC m=+2941.573614215" watchObservedRunningTime="2026-11-02 20:37:23.05953416 +0000 UTC m=+2941.576306746" Nov 02 20:37:23.786339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:23.786310 5800 scope.go:122] "RemoveContainer" containerID="0644e530d4b1e589b2f885fc23979b29a91d686793956b51ba618e24b9c4b2bf" Nov 02 20:37:23.797132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:23.797106 5800 scope.go:122] "RemoveContainer" containerID="d5ed1ea299b181dd8746794ab7a486dee013155bb20cd080b2df2ea11e2eb8db" Nov 02 20:37:23.806798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:23.806774 5800 scope.go:122] "RemoveContainer" containerID="5dbb6f3750fb92977cb2590abb0acfbd2cde4a2de8bf6bfc9e58910b490760bc" Nov 02 20:37:23.958993 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:23.958955 5800 generic.go:356] "Generic (PLEG): container finished" podID="c04c4357-a76e-46b2-9233-60395095c65c" containerID="8b24d3e937d3204735803554c296f4dbc7eb61b3d0820f9af33102b764267b16" exitCode=0 Nov 02 20:37:23.959443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:23.959036 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" event={"ID":"c04c4357-a76e-46b2-9233-60395095c65c","Type":"ContainerDied","Data":"8b24d3e937d3204735803554c296f4dbc7eb61b3d0820f9af33102b764267b16"} Nov 02 20:37:25.250357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.233244 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:25.276811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.276770 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-openshift-ca\") pod \"c04c4357-a76e-46b2-9233-60395095c65c\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " Nov 02 20:37:25.276985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.276824 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04c4357-a76e-46b2-9233-60395095c65c-kube-api-access-2jdbt\" (UniqueName: \"kubernetes.io/projected/c04c4357-a76e-46b2-9233-60395095c65c-kube-api-access-2jdbt\") pod \"c04c4357-a76e-46b2-9233-60395095c65c\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " Nov 02 20:37:25.276985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.276919 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-xks-ca\") pod \"c04c4357-a76e-46b2-9233-60395095c65c\" (UID: \"c04c4357-a76e-46b2-9233-60395095c65c\") " Nov 02 20:37:25.277226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.277200 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-openshift-ca" pod "c04c4357-a76e-46b2-9233-60395095c65c" (UID: "c04c4357-a76e-46b2-9233-60395095c65c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:25.277414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.277396 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:25.279059 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.279038 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04c4357-a76e-46b2-9233-60395095c65c-kube-api-access-2jdbt" pod "c04c4357-a76e-46b2-9233-60395095c65c" (UID: "c04c4357-a76e-46b2-9233-60395095c65c"). InnerVolumeSpecName "kube-api-access-2jdbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:25.279123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.279058 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-xks-ca" pod "c04c4357-a76e-46b2-9233-60395095c65c" (UID: "c04c4357-a76e-46b2-9233-60395095c65c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:37:25.378639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.378606 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4357-a76e-46b2-9233-60395095c65c-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:25.378639 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.378637 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdbt\" (UniqueName: \"kubernetes.io/projected/c04c4357-a76e-46b2-9233-60395095c65c-kube-api-access-2jdbt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:25.976888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.976800 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" Nov 02 20:37:25.976888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.976833 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs" event={"ID":"c04c4357-a76e-46b2-9233-60395095c65c","Type":"ContainerDied","Data":"545ee6ef5cc2a01055cf97bf7e977a4a377e7ccf087445e66e8bc2bab91df55c"} Nov 02 20:37:25.977093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:25.976895 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545ee6ef5cc2a01055cf97bf7e977a4a377e7ccf087445e66e8bc2bab91df55c" Nov 02 20:37:30.012028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:30.011993 5800 generic.go:356] "Generic (PLEG): container finished" podID="22126ebf-024b-45aa-91d1-50488241b3ba" containerID="cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e" exitCode=0 Nov 02 20:37:30.012526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:30.012064 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" event={"ID":"22126ebf-024b-45aa-91d1-50488241b3ba","Type":"ContainerDied","Data":"cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e"} Nov 02 20:37:30.012581 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:30.012531 5800 scope.go:122] "RemoveContainer" containerID="cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e" Nov 02 20:37:30.014131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:30.014105 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:30.014201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:30.014174 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:30.014244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:30.014205 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:31.022345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:31.022307 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" event={"ID":"22126ebf-024b-45aa-91d1-50488241b3ba","Type":"ContainerStarted","Data":"6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702"} Nov 02 20:37:32.018349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:32.018276 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:37:32.019881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:32.019856 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 02 20:37:32.020012 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:32.019916 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 02 20:37:33.017780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:33.017748 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 02 20:37:33.018179 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:33.017802 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 02 20:37:34.017835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:34.017801 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 02 20:37:34.018226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:34.017856 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 02 20:37:34.889839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:34.889802 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:37:35.017529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:35.017488 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 02 20:37:35.017706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:35.017559 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 02 20:37:36.017487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:36.017452 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 02 20:37:36.017888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:36.017503 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 02 20:37:37.022794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:37.022759 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:37:37.023901 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:37.023880 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 20:37:37.043654 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:37.043624 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d"] Nov 02 20:37:37.089807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:37.089776 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13636581-ngr4d"] Nov 02 20:37:37.624438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:37.624401 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be2f913-18b0-439f-ad42-9485d5eeed29" path="/var/lib/kubelet/pods/6be2f913-18b0-439f-ad42-9485d5eeed29/volumes" Nov 02 20:37:42.051752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:42.051710 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt"] Nov 02 20:37:42.097874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:42.097837 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e5da59f-ps8vt"] Nov 02 20:37:43.626743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:43.626695 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c" path="/var/lib/kubelet/pods/ee4a223f-8f4b-4e50-8a2d-07dc52b0df3c/volumes" Nov 02 20:37:49.269171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.269131 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6"] Nov 02 20:37:49.269839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.269820 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c04c4357-a76e-46b2-9233-60395095c65c" containerName="revoke-keys" Nov 02 20:37:49.269886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.269840 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04c4357-a76e-46b2-9233-60395095c65c" containerName="revoke-keys" Nov 02 20:37:49.269984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.269975 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04c4357-a76e-46b2-9233-60395095c65c" containerName="revoke-keys" Nov 02 20:37:49.273630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.273613 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406158 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406112 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406179 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gjv\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-kube-api-access-v6gjv\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406321 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406360 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406430 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406499 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406534 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac776128-7547-4bde-875d-aa7abd33d23b-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406659 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac776128-7547-4bde-875d-aa7abd33d23b-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.406738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.406711 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.438311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.432950 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-09ecd0-openshift-default-dockercfg-ccwlp\"" Nov 02 20:37:49.468799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.468760 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6"] Nov 02 20:37:49.508089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508042 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508125 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508175 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac776128-7547-4bde-875d-aa7abd33d23b-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508238 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac776128-7547-4bde-875d-aa7abd33d23b-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508276 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508320 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508364 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gjv\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-kube-api-access-v6gjv\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508436 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508466 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508560 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-data\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.508927 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.508828 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-credential-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.509408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.509384 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-socket\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.509572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.509436 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-certs\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.509796 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.509769 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac776128-7547-4bde-875d-aa7abd33d23b-istiod-ca-cert\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.512730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.512706 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-envoy\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.513502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.513479 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac776128-7547-4bde-875d-aa7abd33d23b-istio-podinfo\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.562229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.562151 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-istio-token\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.562460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.562438 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gjv\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-kube-api-access-v6gjv\") pod \"e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:49.587808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:49.587775 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:50.013362 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:37:50.013331 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac776128_7547_4bde_875d_aa7abd33d23b.slice/crio-2d62e4db6c43a8aca3de7c9a6cc8d37cd711c65cd154599b699f74de2d8c6c84 WatchSource:0}: Error finding container 2d62e4db6c43a8aca3de7c9a6cc8d37cd711c65cd154599b699f74de2d8c6c84: Status 404 returned error can't find the container with id 2d62e4db6c43a8aca3de7c9a6cc8d37cd711c65cd154599b699f74de2d8c6c84 Nov 02 20:37:50.014018 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.013991 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6"] Nov 02 20:37:50.016233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.016202 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:50.016323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.016267 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:50.016399 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.016383 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:50.205703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.205663 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" event={"ID":"ac776128-7547-4bde-875d-aa7abd33d23b","Type":"ContainerStarted","Data":"e5ddd20311d649b456a492fa3f4ebf58901ec631dfca6da2c6c9ae17a9e5d680"} Nov 02 20:37:50.205884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.205723 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" event={"ID":"ac776128-7547-4bde-875d-aa7abd33d23b","Type":"ContainerStarted","Data":"2d62e4db6c43a8aca3de7c9a6cc8d37cd711c65cd154599b699f74de2d8c6c84"} Nov 02 20:37:50.316886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.316753 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" podStartSLOduration=1.316732211 podStartE2EDuration="1.316732211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:50.313115173 +0000 UTC m=+2968.829887771" watchObservedRunningTime="2026-11-02 20:37:50.316732211 +0000 UTC m=+2968.833504798" Nov 02 20:37:50.588471 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:50.588371 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:51.589984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:51.589947 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 02 20:37:51.590421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:51.589999 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 02 20:37:52.588652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:52.588612 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 02 20:37:52.588875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:52.588673 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 02 20:37:52.696703 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:37:52.696668 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:37:53.588429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:53.588391 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 02 20:37:53.588630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:53.588447 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 02 20:37:54.588417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:54.588378 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 02 20:37:54.588846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:54.588428 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 02 20:37:55.588117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:55.588066 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 02 20:37:55.588326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:55.588153 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 02 20:37:56.593135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:56.593102 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:56.593610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:56.593591 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:56.594429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:56.594409 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:37:57.049883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:57.049845 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs"] Nov 02 20:37:57.053850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:57.053820 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d15d4853-qdrcs"] Nov 02 20:37:57.623892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:37:57.623859 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41a78d3-5375-4d5e-9746-bd27eefc0c27" path="/var/lib/kubelet/pods/d41a78d3-5375-4d5e-9746-bd27eefc0c27/volumes" Nov 02 20:38:08.044389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:08.044292 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5"] Nov 02 20:38:08.089739 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:08.089672 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81d3703-69kx5"] Nov 02 20:38:09.623728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:09.623687 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1825e01e-96de-48a4-a634-e30bfe274475" path="/var/lib/kubelet/pods/1825e01e-96de-48a4-a634-e30bfe274475/volumes" Nov 02 20:38:12.046766 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:12.046726 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5"] Nov 02 20:38:12.092347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:12.092299 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1307b358-sljq5"] Nov 02 20:38:13.623130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:13.623093 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849d08ca-af7f-4c77-99a2-50cf60e06854" path="/var/lib/kubelet/pods/849d08ca-af7f-4c77-99a2-50cf60e06854/volumes" Nov 02 20:38:17.711122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:17.711076 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm"] Nov 02 20:38:19.056307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:19.055353 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz"] Nov 02 20:38:19.057812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:19.057778 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-246579f3-nzlnz"] Nov 02 20:38:19.623760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:19.623731 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7498b75c-7199-4976-90ee-9a234d93a8fb" path="/var/lib/kubelet/pods/7498b75c-7199-4976-90ee-9a234d93a8fb/volumes" Nov 02 20:38:22.183411 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:22.183375 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:38:22.194975 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:22.194950 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:38:22.697260 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:38:22.697227 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:38:22.711803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:22.711750 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" podUID="dfa663b4-496d-4283-8fc8-5d48c9c47913" containerName="payload-processing" containerID="cri-o://10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab" gracePeriod=25 Nov 02 20:38:23.044862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.041801 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:38:23.082957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.082920 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa663b4-496d-4283-8fc8-5d48c9c47913-kube-api-access-c6hrt\" (UniqueName: \"kubernetes.io/projected/dfa663b4-496d-4283-8fc8-5d48c9c47913-kube-api-access-c6hrt\") pod \"dfa663b4-496d-4283-8fc8-5d48c9c47913\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " Nov 02 20:38:23.083134 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.083102 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa663b4-496d-4283-8fc8-5d48c9c47913-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa663b4-496d-4283-8fc8-5d48c9c47913-plugins-config-volume\") pod \"dfa663b4-496d-4283-8fc8-5d48c9c47913\" (UID: \"dfa663b4-496d-4283-8fc8-5d48c9c47913\") " Nov 02 20:38:23.083534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.083506 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa663b4-496d-4283-8fc8-5d48c9c47913-plugins-config-volume" pod "dfa663b4-496d-4283-8fc8-5d48c9c47913" (UID: "dfa663b4-496d-4283-8fc8-5d48c9c47913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:23.083679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.083664 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa663b4-496d-4283-8fc8-5d48c9c47913-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:23.085138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.085110 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa663b4-496d-4283-8fc8-5d48c9c47913-kube-api-access-c6hrt" pod "dfa663b4-496d-4283-8fc8-5d48c9c47913" (UID: "dfa663b4-496d-4283-8fc8-5d48c9c47913"). InnerVolumeSpecName "kube-api-access-c6hrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:23.185421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.185391 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hrt\" (UniqueName: \"kubernetes.io/projected/dfa663b4-496d-4283-8fc8-5d48c9c47913-kube-api-access-c6hrt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:23.512438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.512404 5800 generic.go:356] "Generic (PLEG): container finished" podID="dfa663b4-496d-4283-8fc8-5d48c9c47913" containerID="10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab" exitCode=0 Nov 02 20:38:23.512650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.512456 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" Nov 02 20:38:23.512650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.512483 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" event={"ID":"dfa663b4-496d-4283-8fc8-5d48c9c47913","Type":"ContainerDied","Data":"10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab"} Nov 02 20:38:23.512650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.512566 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm" event={"ID":"dfa663b4-496d-4283-8fc8-5d48c9c47913","Type":"ContainerDied","Data":"14969ede112dc12701e5b5cd0c528029d9555a184ce6cb10b17fc2db9f55f3e9"} Nov 02 20:38:23.512650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.512602 5800 scope.go:122] "RemoveContainer" containerID="10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab" Nov 02 20:38:23.529900 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.529872 5800 scope.go:122] "RemoveContainer" containerID="10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab" Nov 02 20:38:23.530973 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:38:23.530943 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab\": container with ID starting with 10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab not found: ID does not exist" containerID="10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab" Nov 02 20:38:23.531105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.530983 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab"} err="failed to get container status \"10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab\": rpc error: code = NotFound desc = could not find container \"10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab\": container with ID starting with 10605a95ca6206b563f103e49da4d35c43cfd7004af5ea87d3b0cad9c1b702ab not found: ID does not exist" Nov 02 20:38:23.672638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.672594 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm"] Nov 02 20:38:23.676133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.676102 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-920071-84d8ffd7c9-2nmkm"] Nov 02 20:38:23.874115 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.874025 5800 scope.go:122] "RemoveContainer" containerID="f4b63b4a72514802a822e69ab6090ad5220aacc724d9387326a32b3a6efa0a6b" Nov 02 20:38:23.885736 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.885713 5800 scope.go:122] "RemoveContainer" containerID="b0894df2fe1077e52329f5e23cb652fc29f744c4fab104e5e2709005cdd12555" Nov 02 20:38:23.895887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.895861 5800 scope.go:122] "RemoveContainer" containerID="0ea39df8a690180644c7f872a20412d1f8dafa4fdb9c3171ee0b9ea8f23e25a7" Nov 02 20:38:23.905573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.905549 5800 scope.go:122] "RemoveContainer" containerID="95c59f7dba9ffa9e3a9fb88855a0b171a67698ffa42ef3d58ebb81988f664e52" Nov 02 20:38:23.915210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.915187 5800 scope.go:122] "RemoveContainer" containerID="1b0ba8ce401422ff1f4e531ad4e03c954eddd4dfbd47e3dcfe20c0bc7ed05b1c" Nov 02 20:38:23.924333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.924272 5800 scope.go:122] "RemoveContainer" containerID="fb7d9c197be13dec897fb21fb57460361738dacd56e9d6002618988b6b7fb7b0" Nov 02 20:38:23.934067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:23.934044 5800 scope.go:122] "RemoveContainer" containerID="a5ca74226ef4c33d94c87f251220e8f7ffa65151f7f07e478d0be3ac5d12a992" Nov 02 20:38:24.047638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:24.047608 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz"] Nov 02 20:38:24.052663 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:24.052621 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k"] Nov 02 20:38:24.055900 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:24.055876 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6641197c-s67hz"] Nov 02 20:38:24.094259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:24.094225 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2a0397-5wj6k"] Nov 02 20:38:25.624914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:25.624880 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035eb817-2660-479b-a30f-50997f181af6" path="/var/lib/kubelet/pods/035eb817-2660-479b-a30f-50997f181af6/volumes" Nov 02 20:38:25.625404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:25.625385 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f4b91d-b18c-492b-9fcb-0096a1c08515" path="/var/lib/kubelet/pods/14f4b91d-b18c-492b-9fcb-0096a1c08515/volumes" Nov 02 20:38:25.625756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:25.625742 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa663b4-496d-4283-8fc8-5d48c9c47913" path="/var/lib/kubelet/pods/dfa663b4-496d-4283-8fc8-5d48c9c47913/volumes" Nov 02 20:38:34.715248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.715216 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m"] Nov 02 20:38:34.715856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.715842 5800 cpu_manager.go:414] "containerMap: removing container" podUID="dfa663b4-496d-4283-8fc8-5d48c9c47913" containerName="payload-processing" Nov 02 20:38:34.715900 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.715857 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa663b4-496d-4283-8fc8-5d48c9c47913" containerName="payload-processing" Nov 02 20:38:34.715988 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.715979 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa663b4-496d-4283-8fc8-5d48c9c47913" containerName="payload-processing" Nov 02 20:38:34.721709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.721687 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.771057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.771023 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m"] Nov 02 20:38:34.781803 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.781769 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.782096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.782063 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-kube-api-access-tdxwm\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.782200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.782120 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.813699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.813667 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:38:34.814660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.814018 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:38:34.814660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.814133 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:38:34.816453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.816411 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:38:34.883450 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.883416 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-kube-api-access-tdxwm\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.883662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.883472 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.883767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.883735 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.884381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.884355 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.886825 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.885951 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:34.960903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:34.960866 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-kube-api-access-tdxwm\") pod \"maas-api-revoke-sub-981bf270-4cg6m\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:35.060726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:35.060677 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:35.461373 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:38:35.461307 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725d3bfb_81cd_4da8_9f9d_6e598206b4ea.slice/crio-dc7e73eee92210e2e59b15ad3c404c34a0ec6bbab23265c3593d874f125bc84b WatchSource:0}: Error finding container dc7e73eee92210e2e59b15ad3c404c34a0ec6bbab23265c3593d874f125bc84b: Status 404 returned error can't find the container with id dc7e73eee92210e2e59b15ad3c404c34a0ec6bbab23265c3593d874f125bc84b Nov 02 20:38:35.461373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:35.461340 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m"] Nov 02 20:38:35.680323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:35.680266 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" event={"ID":"725d3bfb-81cd-4da8-9f9d-6e598206b4ea","Type":"ContainerStarted","Data":"dc7e73eee92210e2e59b15ad3c404c34a0ec6bbab23265c3593d874f125bc84b"} Nov 02 20:38:36.056324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:36.056267 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm"] Nov 02 20:38:36.094414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:36.094362 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344bb3a5-x7hlm"] Nov 02 20:38:36.689522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:36.689484 5800 generic.go:356] "Generic (PLEG): container finished" podID="725d3bfb-81cd-4da8-9f9d-6e598206b4ea" containerID="54e9bfa1bdade7189180a110442e43edb7d11b2b8c3acc9b6c9bb59b5e28332f" exitCode=0 Nov 02 20:38:36.689679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:36.689558 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" event={"ID":"725d3bfb-81cd-4da8-9f9d-6e598206b4ea","Type":"ContainerDied","Data":"54e9bfa1bdade7189180a110442e43edb7d11b2b8c3acc9b6c9bb59b5e28332f"} Nov 02 20:38:37.664606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:37.664567 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e" path="/var/lib/kubelet/pods/b4e37ad8-6e52-4bb2-bc4f-84db70d6b38e/volumes" Nov 02 20:38:37.912809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:37.912783 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:38.060339 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.059878 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-xks-ca\") pod \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " Nov 02 20:38:38.060773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.060749 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-openshift-ca\") pod \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " Nov 02 20:38:38.061394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.061374 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-kube-api-access-tdxwm\") pod \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\" (UID: \"725d3bfb-81cd-4da8-9f9d-6e598206b4ea\") " Nov 02 20:38:38.062334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.061223 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-openshift-ca" pod "725d3bfb-81cd-4da8-9f9d-6e598206b4ea" (UID: "725d3bfb-81cd-4da8-9f9d-6e598206b4ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:38:38.067352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.065418 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-xks-ca" pod "725d3bfb-81cd-4da8-9f9d-6e598206b4ea" (UID: "725d3bfb-81cd-4da8-9f9d-6e598206b4ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:38:38.067604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.067584 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-kube-api-access-tdxwm" pod "725d3bfb-81cd-4da8-9f9d-6e598206b4ea" (UID: "725d3bfb-81cd-4da8-9f9d-6e598206b4ea"). InnerVolumeSpecName "kube-api-access-tdxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:38:38.163502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.163467 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdxwm\" (UniqueName: \"kubernetes.io/projected/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-kube-api-access-tdxwm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:38.163502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.163499 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:38.163502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.163509 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725d3bfb-81cd-4da8-9f9d-6e598206b4ea-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:38:38.706558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.706526 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" Nov 02 20:38:38.706967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.706553 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m" event={"ID":"725d3bfb-81cd-4da8-9f9d-6e598206b4ea","Type":"ContainerDied","Data":"dc7e73eee92210e2e59b15ad3c404c34a0ec6bbab23265c3593d874f125bc84b"} Nov 02 20:38:38.706967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:38.706605 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7e73eee92210e2e59b15ad3c404c34a0ec6bbab23265c3593d874f125bc84b" Nov 02 20:38:42.746764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:42.746662 5800 generic.go:356] "Generic (PLEG): container finished" podID="32638407-68f5-4686-b8f4-3367e9009481" containerID="d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1" exitCode=0 Nov 02 20:38:42.746764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:42.746706 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" event={"ID":"32638407-68f5-4686-b8f4-3367e9009481","Type":"ContainerDied","Data":"d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1"} Nov 02 20:38:42.747414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:42.747166 5800 scope.go:122] "RemoveContainer" containerID="d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1" Nov 02 20:38:42.748947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:42.748913 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:42.749055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:42.749002 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:42.749055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:42.749044 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:43.758829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:43.758795 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" event={"ID":"32638407-68f5-4686-b8f4-3367e9009481","Type":"ContainerStarted","Data":"1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9"} Nov 02 20:38:44.545308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:44.542742 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:38:44.713252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:44.713215 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:38:44.714636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:44.714610 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 02 20:38:44.714756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:44.714655 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 02 20:38:45.713180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:45.713145 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 02 20:38:45.713677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:45.713202 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 02 20:38:46.713566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:46.713535 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 02 20:38:46.713966 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:46.713591 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 02 20:38:47.056706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:47.056662 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4"] Nov 02 20:38:47.098320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:47.096167 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16413107-mmqd4"] Nov 02 20:38:47.671318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:47.670188 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc52b400-a81d-48cd-9323-fe8348a9a054" path="/var/lib/kubelet/pods/fc52b400-a81d-48cd-9323-fe8348a9a054/volumes" Nov 02 20:38:47.726304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:47.725779 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 02 20:38:47.726304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:47.725848 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 02 20:38:48.713082 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:48.713041 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Nov 02 20:38:48.713274 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:48.713109 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Nov 02 20:38:49.717528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:49.717496 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:38:49.718867 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:49.718826 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 20:38:52.695990 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:38:52.695953 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:38:58.044412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:58.044375 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss"] Nov 02 20:38:58.089767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:58.089733 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbd0381f-s25ss"] Nov 02 20:38:59.624108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:38:59.624069 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa74b8d1-9afa-4e9d-994a-5eaea0181490" path="/var/lib/kubelet/pods/fa74b8d1-9afa-4e9d-994a-5eaea0181490/volumes" Nov 02 20:39:22.699668 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:39:22.699624 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:39:24.002121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.002082 5800 scope.go:122] "RemoveContainer" containerID="ae25bc5bdc5108628f54735b0396de77491a05a02e79d06d0360edf7ab255ac2" Nov 02 20:39:24.012703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.012681 5800 scope.go:122] "RemoveContainer" containerID="dd4f550f9b02c9db26a4ea9a5c08f5bbc32e9b32cffc0a44f0c313f99223b6ec" Nov 02 20:39:24.022310 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.022269 5800 scope.go:122] "RemoveContainer" containerID="4daaf79403c3ba73dc8aeecea31d20b2b27328800f4b0bbb329d9122c1f0ea98" Nov 02 20:39:24.047696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.047662 5800 scope.go:122] "RemoveContainer" containerID="ff2ad897804a6ca9d659a7fa51a65be9f3eba1b2b00183f8e60bd32e0be2e5f8" Nov 02 20:39:24.097491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.097464 5800 scope.go:122] "RemoveContainer" containerID="bfde71d4cf6cfb0aaf4c33a2c4aba2683c307e54d6ae0c6f4b1acd32ede4c014" Nov 02 20:39:24.108174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.108154 5800 scope.go:122] "RemoveContainer" containerID="2c2cc9a9c17df94d91ffc825a10cc0467d39d3dd341cddbb041723b671ae7a88" Nov 02 20:39:24.117740 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.117719 5800 scope.go:122] "RemoveContainer" containerID="59fd5573f073b5312efcf7e98f9e67bcf43f1025c985e16958f78b6c182383ee" Nov 02 20:39:24.127533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:24.127513 5800 scope.go:122] "RemoveContainer" containerID="1fe05f367f252cba99ac701d3a5197819281faa5fa295e3b24bc459fda49cdf8" Nov 02 20:39:28.754637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:28.754604 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2"] Nov 02 20:39:28.755264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:28.754825 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" containerID="cri-o://d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d" gracePeriod=30 Nov 02 20:39:28.801425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:28.801388 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl"] Nov 02 20:39:28.801660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:28.801634 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" containerID="cri-o://c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58" gracePeriod=30 Nov 02 20:39:28.805017 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:28.804989 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g"] Nov 02 20:39:28.805248 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:28.805209 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" containerID="cri-o://7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b" gracePeriod=30 Nov 02 20:39:34.111381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.111347 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:39:34.145551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.145526 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:39:34.149168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.149143 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:39:34.179875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.179842 5800 generic.go:356] "Generic (PLEG): container finished" podID="049812e6-2949-4744-9381-3b3ab5290247" containerID="d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d" exitCode=0 Nov 02 20:39:34.180036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.179890 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" Nov 02 20:39:34.180036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.179934 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" event={"ID":"049812e6-2949-4744-9381-3b3ab5290247","Type":"ContainerDied","Data":"d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d"} Nov 02 20:39:34.180036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.179993 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2" event={"ID":"049812e6-2949-4744-9381-3b3ab5290247","Type":"ContainerDied","Data":"3a96d3bbc1976f1cdf0d9394605bd964d4d311b54299ab1f319dfb2ef96c95ed"} Nov 02 20:39:34.180036 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.180016 5800 scope.go:122] "RemoveContainer" containerID="d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d" Nov 02 20:39:34.181467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.181443 5800 generic.go:356] "Generic (PLEG): container finished" podID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerID="c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58" exitCode=0 Nov 02 20:39:34.181589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.181486 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" Nov 02 20:39:34.181589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.181506 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" event={"ID":"b1ff5090-175b-45c7-9800-aee8c7cd57f1","Type":"ContainerDied","Data":"c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58"} Nov 02 20:39:34.181717 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.181588 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl" event={"ID":"b1ff5090-175b-45c7-9800-aee8c7cd57f1","Type":"ContainerDied","Data":"c2f727ed7f6eb96c97aeda2edf767817f8f940156ac69c5a4427149b09adc9c7"} Nov 02 20:39:34.182886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.182867 5800 generic.go:356] "Generic (PLEG): container finished" podID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerID="7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b" exitCode=0 Nov 02 20:39:34.183015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.182906 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" Nov 02 20:39:34.183015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.182949 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" event={"ID":"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238","Type":"ContainerDied","Data":"7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b"} Nov 02 20:39:34.183015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.182990 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g" event={"ID":"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238","Type":"ContainerDied","Data":"ac8a1029f39678b7b9f7b75fd16dd80c28c0e6eae055da7f6459fa124a764eee"} Nov 02 20:39:34.190951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.190929 5800 scope.go:122] "RemoveContainer" containerID="d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d" Nov 02 20:39:34.191247 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:39:34.191227 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d\": container with ID starting with d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d not found: ID does not exist" containerID="d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d" Nov 02 20:39:34.191356 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.191255 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d"} err="failed to get container status \"d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d\": rpc error: code = NotFound desc = could not find container \"d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d\": container with ID starting with d5f862c143a13e5f35611c5ff6fee7a12427b06a9e7bd1042b9283c57bce028d not found: ID does not exist" Nov 02 20:39:34.191356 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.191273 5800 scope.go:122] "RemoveContainer" containerID="c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58" Nov 02 20:39:34.202363 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.202344 5800 scope.go:122] "RemoveContainer" containerID="c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58" Nov 02 20:39:34.202649 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:39:34.202632 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58\": container with ID starting with c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58 not found: ID does not exist" containerID="c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58" Nov 02 20:39:34.202691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.202657 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58"} err="failed to get container status \"c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58\": rpc error: code = NotFound desc = could not find container \"c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58\": container with ID starting with c96d817957d6448a34858ee7b473eb247c42e553667a7cfad23ec5139b9bdf58 not found: ID does not exist" Nov 02 20:39:34.202691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.202674 5800 scope.go:122] "RemoveContainer" containerID="7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b" Nov 02 20:39:34.212091 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.212074 5800 scope.go:122] "RemoveContainer" containerID="7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b" Nov 02 20:39:34.212424 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:39:34.212405 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b\": container with ID starting with 7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b not found: ID does not exist" containerID="7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b" Nov 02 20:39:34.212481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.212434 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b"} err="failed to get container status \"7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b\": rpc error: code = NotFound desc = could not find container \"7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b\": container with ID starting with 7fcbdf53b0035cc064afd30074c93af960eb5d940b66e7252f81e705faadc65b not found: ID does not exist" Nov 02 20:39:34.216800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216780 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-data\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.216888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216810 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-podinfo\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.216888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216828 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-socket\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.216888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216848 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-envoy\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.216888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216880 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-certs\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.217105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216914 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istiod-ca-cert\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.217105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216942 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-data\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.217105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.216981 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-certs\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.217105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217024 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-envoy\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.217105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217057 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-podinfo\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.217105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217075 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-data" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.217105 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217087 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049812e6-2949-4744-9381-3b3ab5290247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049812e6-2949-4744-9381-3b3ab5290247-istiod-ca-cert\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217107 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-socket" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217136 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/049812e6-2949-4744-9381-3b3ab5290247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049812e6-2949-4744-9381-3b3ab5290247-istio-podinfo\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217145 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-certs" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217188 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-socket\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217224 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-kube-api-access-lv59v\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-kube-api-access-lv59v\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217312 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-certs" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217336 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-credential-socket\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217377 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-socket\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217415 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-credential-socket\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217450 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-istio-token\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-istio-token\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217487 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-envoy\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217517 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-socket" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.217532 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217524 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-kube-api-access-hvd6w\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-kube-api-access-hvd6w\") pod \"049812e6-2949-4744-9381-3b3ab5290247\" (UID: \"049812e6-2949-4744-9381-3b3ab5290247\") " Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217557 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049812e6-2949-4744-9381-3b3ab5290247-istiod-ca-cert" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217563 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istiod-ca-cert" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217603 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-token\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-token\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217818 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-credential-socket" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.217235 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-data" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218013 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-socket" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218105 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-certs\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218178 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-kube-api-access-824qx\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-kube-api-access-824qx\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.218254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218220 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-data\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.218767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218321 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istiod-ca-cert\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.218767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218392 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-token\") pod \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\" (UID: \"b1ff5090-175b-45c7-9800-aee8c7cd57f1\") " Nov 02 20:39:34.218767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218442 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-credential-socket\") pod \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\" (UID: \"d8d4bf32-6cbb-4d68-aee2-265b1c9eb238\") " Nov 02 20:39:34.218767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218447 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-credential-socket" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.218767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218711 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-data" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.218767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218712 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-certs" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.219114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218816 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istiod-ca-cert" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:34.219114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.218931 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-credential-socket" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.219235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219215 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219241 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219257 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219272 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219301 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219314 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219328 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219341 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219357 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049812e6-2949-4744-9381-3b3ab5290247-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219374 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219388 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219402 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219416 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219430 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219443 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.219799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.219722 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-envoy" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.220556 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.220514 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-envoy" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.221412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.221242 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-podinfo" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:39:34.221535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.221488 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-istio-token" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:34.221535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.221528 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-kube-api-access-lv59v" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "kube-api-access-lv59v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:34.221735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.221692 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-kube-api-access-hvd6w" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "kube-api-access-hvd6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:34.221953 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.221931 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-envoy" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:34.222022 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.221999 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-token" pod "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" (UID: "d8d4bf32-6cbb-4d68-aee2-265b1c9eb238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:34.222122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.222105 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-podinfo" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:39:34.222184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.222169 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/049812e6-2949-4744-9381-3b3ab5290247-istio-podinfo" pod "049812e6-2949-4744-9381-3b3ab5290247" (UID: "049812e6-2949-4744-9381-3b3ab5290247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:39:34.222622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.222600 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-kube-api-access-824qx" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "kube-api-access-824qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:34.222726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.222677 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-token" pod "b1ff5090-175b-45c7-9800-aee8c7cd57f1" (UID: "b1ff5090-175b-45c7-9800-aee8c7cd57f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:34.321309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321206 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321247 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321262 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321296 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049812e6-2949-4744-9381-3b3ab5290247-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321310 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321325 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049812e6-2949-4744-9381-3b3ab5290247-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321339 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv59v\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-kube-api-access-lv59v\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321353 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321367 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ff5090-175b-45c7-9800-aee8c7cd57f1-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321380 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvd6w\" (UniqueName: \"kubernetes.io/projected/049812e6-2949-4744-9381-3b3ab5290247-kube-api-access-hvd6w\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321395 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.321596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.321409 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824qx\" (UniqueName: \"kubernetes.io/projected/b1ff5090-175b-45c7-9800-aee8c7cd57f1-kube-api-access-824qx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:34.656291 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.656179 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2"] Nov 02 20:39:34.660111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.660080 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-v6mc2"] Nov 02 20:39:34.760443 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.760411 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g"] Nov 02 20:39:34.762037 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.762015 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-5zb2g"] Nov 02 20:39:34.864433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.864380 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl"] Nov 02 20:39:34.866982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:34.866952 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-426941-openshift-default-6f5c885b5-w49vl"] Nov 02 20:39:35.622908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:35.622872 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049812e6-2949-4744-9381-3b3ab5290247" path="/var/lib/kubelet/pods/049812e6-2949-4744-9381-3b3ab5290247/volumes" Nov 02 20:39:35.623472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:35.623456 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" path="/var/lib/kubelet/pods/b1ff5090-175b-45c7-9800-aee8c7cd57f1/volumes" Nov 02 20:39:35.623910 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:35.623897 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" path="/var/lib/kubelet/pods/d8d4bf32-6cbb-4d68-aee2-265b1c9eb238/volumes" Nov 02 20:39:52.697509 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:39:52.697467 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:39:53.085165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:53.085124 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth"] Nov 02 20:39:53.085438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:53.085410 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" containerID="cri-o://232afaece1508610044454adec01fb7a895db07f0d13b19f5942e2df5e5ca347" gracePeriod=30 Nov 02 20:39:53.131470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:53.131432 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t"] Nov 02 20:39:53.131675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:53.131651 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" containerID="cri-o://6e5d5a17331d7ac84ee430d84c05d4b0d7d1adcd6274679aea5fd816c9a39e5d" gracePeriod=30 Nov 02 20:39:58.370418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.370339 5800 generic.go:356] "Generic (PLEG): container finished" podID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerID="232afaece1508610044454adec01fb7a895db07f0d13b19f5942e2df5e5ca347" exitCode=0 Nov 02 20:39:58.370877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.370420 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" event={"ID":"ab0ba614-2d5d-493c-9eb2-a6e648f657ab","Type":"ContainerDied","Data":"232afaece1508610044454adec01fb7a895db07f0d13b19f5942e2df5e5ca347"} Nov 02 20:39:58.372491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.372469 5800 generic.go:356] "Generic (PLEG): container finished" podID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerID="6e5d5a17331d7ac84ee430d84c05d4b0d7d1adcd6274679aea5fd816c9a39e5d" exitCode=0 Nov 02 20:39:58.372622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.372546 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" event={"ID":"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed","Type":"ContainerDied","Data":"6e5d5a17331d7ac84ee430d84c05d4b0d7d1adcd6274679aea5fd816c9a39e5d"} Nov 02 20:39:58.420748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.420724 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:39:58.452651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.452616 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:39:58.541345 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541302 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-envoy\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.541554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541365 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-envoy\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.541554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541396 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-token\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-token\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.541554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541416 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-socket\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.541554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541477 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-socket\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.541554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541518 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-data\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541570 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-kube-api-access-dp6p4\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-kube-api-access-dp6p4\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541619 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-podinfo\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541655 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-kube-api-access-lv6fp\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-kube-api-access-lv6fp\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541684 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-certs\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541709 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-podinfo\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541757 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-certs\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541805 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-socket" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.541845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541823 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-credential-socket\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.542276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541917 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-token\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.542276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.541973 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-credential-socket\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.542276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.542058 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istiod-ca-cert\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.542276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.542117 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-data\") pod \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\" (UID: \"ab0ba614-2d5d-493c-9eb2-a6e648f657ab\") " Nov 02 20:39:58.542276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.542146 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istiod-ca-cert\") pod \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\" (UID: \"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed\") " Nov 02 20:39:58.542986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.542055 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-credential-socket" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.542986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.542809 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-data" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.543143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543011 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-credential-socket" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.543200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543170 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.543258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543201 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-certs" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.543418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543383 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-socket" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.543572 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543547 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-data" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.543657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543553 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-certs" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.543657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543632 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istiod-ca-cert" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:58.543771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.543674 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istiod-ca-cert" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:58.544350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.544321 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-envoy" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.544550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.544528 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-envoy" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:39:58.545782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.545749 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-token" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:58.545890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.545853 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-podinfo" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:39:58.546058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.546030 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-kube-api-access-lv6fp" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "kube-api-access-lv6fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:58.546323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.546272 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-podinfo" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:39:58.546595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.546578 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-kube-api-access-dp6p4" pod "ab0ba614-2d5d-493c-9eb2-a6e648f657ab" (UID: "ab0ba614-2d5d-493c-9eb2-a6e648f657ab"). InnerVolumeSpecName "kube-api-access-dp6p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:58.547131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.547107 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-token" pod "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" (UID: "5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644678 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644711 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644722 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644731 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp6p4\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-kube-api-access-dp6p4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644743 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644752 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv6fp\" (UniqueName: \"kubernetes.io/projected/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-kube-api-access-lv6fp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644760 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644769 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.644774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644779 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644787 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644797 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644806 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644814 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644822 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644830 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644838 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:58.645424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:58.644847 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0ba614-2d5d-493c-9eb2-a6e648f657ab-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:59.381506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.381424 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" Nov 02 20:39:59.381506 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.381464 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t" event={"ID":"5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed","Type":"ContainerDied","Data":"fd8c2e269d8df362c0b1d68a296b4a6fe41fc3dd4e9313ff4492aaded4e74af2"} Nov 02 20:39:59.382026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.381532 5800 scope.go:122] "RemoveContainer" containerID="6e5d5a17331d7ac84ee430d84c05d4b0d7d1adcd6274679aea5fd816c9a39e5d" Nov 02 20:39:59.383223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.383196 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" event={"ID":"ab0ba614-2d5d-493c-9eb2-a6e648f657ab","Type":"ContainerDied","Data":"04fcc0c2767731c5e175a635f49fb15a16406b90bc44de253514be9b97fd1e2c"} Nov 02 20:39:59.383373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.383234 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth" Nov 02 20:39:59.393954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.393935 5800 scope.go:122] "RemoveContainer" containerID="232afaece1508610044454adec01fb7a895db07f0d13b19f5942e2df5e5ca347" Nov 02 20:39:59.543578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.543543 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth"] Nov 02 20:39:59.589810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.589773 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-t4jth"] Nov 02 20:39:59.623039 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.623003 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" path="/var/lib/kubelet/pods/ab0ba614-2d5d-493c-9eb2-a6e648f657ab/volumes" Nov 02 20:39:59.652880 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.652799 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t"] Nov 02 20:39:59.660741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:39:59.660709 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-x4m2t"] Nov 02 20:40:01.625891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:01.625851 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" path="/var/lib/kubelet/pods/5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed/volumes" Nov 02 20:40:19.257651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.257611 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z"] Nov 02 20:40:19.258519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258497 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" Nov 02 20:40:19.258600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258521 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" Nov 02 20:40:19.258600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258547 5800 cpu_manager.go:414] "containerMap: removing container" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" Nov 02 20:40:19.258600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258554 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" Nov 02 20:40:19.258600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258587 5800 cpu_manager.go:414] "containerMap: removing container" podUID="725d3bfb-81cd-4da8-9f9d-6e598206b4ea" containerName="revoke-keys" Nov 02 20:40:19.258600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258597 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725d3bfb-81cd-4da8-9f9d-6e598206b4ea" containerName="revoke-keys" Nov 02 20:40:19.258767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258636 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" Nov 02 20:40:19.258767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258644 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" Nov 02 20:40:19.258767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258664 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" Nov 02 20:40:19.258767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258671 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" Nov 02 20:40:19.258767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258689 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" Nov 02 20:40:19.258767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258696 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" Nov 02 20:40:19.258952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258823 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4994ee-a43f-4eb7-ae76-44ffdf28a9ed" containerName="istio-proxy" Nov 02 20:40:19.258952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258837 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0ba614-2d5d-493c-9eb2-a6e648f657ab" containerName="istio-proxy" Nov 02 20:40:19.258952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258845 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d4bf32-6cbb-4d68-aee2-265b1c9eb238" containerName="istio-proxy" Nov 02 20:40:19.258952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258856 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="049812e6-2949-4744-9381-3b3ab5290247" containerName="istio-proxy" Nov 02 20:40:19.258952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258871 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="725d3bfb-81cd-4da8-9f9d-6e598206b4ea" containerName="revoke-keys" Nov 02 20:40:19.258952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.258884 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ff5090-175b-45c7-9800-aee8c7cd57f1" containerName="istio-proxy" Nov 02 20:40:19.264454 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.264434 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.312584 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.312552 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z"] Nov 02 20:40:19.325033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.324998 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-tmp-dir\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.325193 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.325062 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-home\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.325243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.325204 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772652a0-a403-42fe-a14a-adb11615b9ec-tls-certs\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.325315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.325265 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-model-cache\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.325373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.325332 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-dshm\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.325469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.325448 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwm6\" (UniqueName: \"kubernetes.io/projected/772652a0-a403-42fe-a14a-adb11615b9ec-kube-api-access-rtwm6\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.332007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.331983 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"openshift-service-ca.crt\"" Nov 02 20:40:19.354751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.354705 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"kube-root-ca.crt\"" Nov 02 20:40:19.354901 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.354779 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"auth-test-model-a9ed60-kserve-self-signed-certs\"" Nov 02 20:40:19.360016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.359997 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a9ed60\"/\"default-dockercfg-bjw6w\"" Nov 02 20:40:19.427150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.427115 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-tmp-dir\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.427402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.427181 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-home\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.427402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.427270 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772652a0-a403-42fe-a14a-adb11615b9ec-tls-certs\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.427402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.427346 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-model-cache\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.427402 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.427378 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-dshm\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.427677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.427470 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwm6\" (UniqueName: \"kubernetes.io/projected/772652a0-a403-42fe-a14a-adb11615b9ec-kube-api-access-rtwm6\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.428225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.428112 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-tmp-dir\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.428425 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.428404 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-home\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.430118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.430098 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-model-cache\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.434176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.434151 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-dshm\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.436256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.436233 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772652a0-a403-42fe-a14a-adb11615b9ec-tls-certs\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.480968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.480928 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwm6\" (UniqueName: \"kubernetes.io/projected/772652a0-a403-42fe-a14a-adb11615b9ec-kube-api-access-rtwm6\") pod \"auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.580528 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.580439 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:19.958467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:19.958437 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z"] Nov 02 20:40:19.960004 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:40:19.959974 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod772652a0_a403_42fe_a14a_adb11615b9ec.slice/crio-bb5da0c398c718c29e0b6d094adb80f4accd028efe1cd7d45ad8a8b0f95d3601 WatchSource:0}: Error finding container bb5da0c398c718c29e0b6d094adb80f4accd028efe1cd7d45ad8a8b0f95d3601: Status 404 returned error can't find the container with id bb5da0c398c718c29e0b6d094adb80f4accd028efe1cd7d45ad8a8b0f95d3601 Nov 02 20:40:20.540328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:20.540276 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" event={"ID":"772652a0-a403-42fe-a14a-adb11615b9ec","Type":"ContainerStarted","Data":"c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92"} Nov 02 20:40:20.540954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:20.540342 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" event={"ID":"772652a0-a403-42fe-a14a-adb11615b9ec","Type":"ContainerStarted","Data":"bb5da0c398c718c29e0b6d094adb80f4accd028efe1cd7d45ad8a8b0f95d3601"} Nov 02 20:40:20.580747 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:20.580710 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:20.646004 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:20.645943 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" podStartSLOduration=1.645924318 podStartE2EDuration="1.645924318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:40:20.645381406 +0000 UTC m=+3119.162153992" watchObservedRunningTime="2026-11-02 20:40:20.645924318 +0000 UTC m=+3119.162696904" Nov 02 20:40:22.698078 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:40:22.698029 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:40:24.193945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:24.193914 5800 scope.go:122] "RemoveContainer" containerID="976e526439aaf7e80e70e0e5f225a46fa25911a112efccd66aac8348d06efd58" Nov 02 20:40:24.204064 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:24.204041 5800 scope.go:122] "RemoveContainer" containerID="36d0c26a327811259b52cc7a68edff7846bcf2f744d5acb6c84e320a2c8e79a0" Nov 02 20:40:24.213726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:24.213707 5800 scope.go:122] "RemoveContainer" containerID="b2ef480c885398d9653515d86fca95df050a15c60fea6c3d1c005408e0dda94b" Nov 02 20:40:29.581359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:29.581317 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:29.595096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:29.595061 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:29.655696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:40:29.655664 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:40:52.696666 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:40:52.696628 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:41:22.697346 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:41:22.697252 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:41:52.698789 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:41:52.698753 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:42:22.696252 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:42:22.696216 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:42:26.044456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:42:26.044418 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs"] Nov 02 20:42:26.091089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:42:26.091046 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b506af-znhjs"] Nov 02 20:42:27.622710 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:42:27.622666 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04c4357-a76e-46b2-9233-60395095c65c" path="/var/lib/kubelet/pods/c04c4357-a76e-46b2-9233-60395095c65c/volumes" Nov 02 20:42:52.695849 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:42:52.695811 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:43:22.248516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:22.248484 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:43:22.266196 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:22.266164 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:43:22.698720 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:43:22.698682 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:43:24.297169 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:24.297090 5800 scope.go:122] "RemoveContainer" containerID="8b24d3e937d3204735803554c296f4dbc7eb61b3d0820f9af33102b764267b16" Nov 02 20:43:39.045213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:39.045132 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m"] Nov 02 20:43:39.090604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:39.090569 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981bf270-4cg6m"] Nov 02 20:43:39.622782 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:39.622746 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725d3bfb-81cd-4da8-9f9d-6e598206b4ea" path="/var/lib/kubelet/pods/725d3bfb-81cd-4da8-9f9d-6e598206b4ea/volumes" Nov 02 20:43:49.329912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.329877 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j"] Nov 02 20:43:49.334138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.334120 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.462892 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.462854 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-credential-socket\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.462899 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-token\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.462996 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h4tj\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-kube-api-access-5h4tj\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.463018 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-data\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.463036 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-envoy\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.463064 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-socket\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.463116 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb290aa-6a87-4d60-94f1-412eba22be22-istiod-ca-cert\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.463178 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-podinfo\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.463496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.463203 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-certs\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.465812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.465134 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-09e014-openshift-default-dockercfg-csdqz\"" Nov 02 20:43:49.471185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.471147 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j"] Nov 02 20:43:49.564711 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.564672 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-podinfo\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.564933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.564717 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-certs\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.564933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.564767 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-credential-socket\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.564933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.564794 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-token\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.564933 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.564923 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h4tj\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-kube-api-access-5h4tj\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.565162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.564946 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-data\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.565162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.564975 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-envoy\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.565162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.565020 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-socket\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.565162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.565089 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb290aa-6a87-4d60-94f1-412eba22be22-istiod-ca-cert\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.565447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.565264 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-certs\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.566096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.565540 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-data\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.566096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.565891 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-socket\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.566096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.565899 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-credential-socket\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.566096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.565922 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb290aa-6a87-4d60-94f1-412eba22be22-istiod-ca-cert\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.567998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.567973 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-podinfo\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.568135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.568052 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-envoy\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.618265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.618190 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-token\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.618446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.618356 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h4tj\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-kube-api-access-5h4tj\") pod \"e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:49.648407 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:49.648370 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:50.077769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:50.077741 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j"] Nov 02 20:43:50.078644 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:43:50.078615 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb290aa_6a87_4d60_94f1_412eba22be22.slice/crio-af55a5b7fa93476ac7297a872e77756034aa05938a6781a0015a7d82bb6c5d47 WatchSource:0}: Error finding container af55a5b7fa93476ac7297a872e77756034aa05938a6781a0015a7d82bb6c5d47: Status 404 returned error can't find the container with id af55a5b7fa93476ac7297a872e77756034aa05938a6781a0015a7d82bb6c5d47 Nov 02 20:43:50.081153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:50.081119 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:50.081270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:50.081197 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:50.081270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:50.081242 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:50.162362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:50.162324 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" event={"ID":"2bb290aa-6a87-4d60-94f1-412eba22be22","Type":"ContainerStarted","Data":"af55a5b7fa93476ac7297a872e77756034aa05938a6781a0015a7d82bb6c5d47"} Nov 02 20:43:51.171918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:51.171876 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" event={"ID":"2bb290aa-6a87-4d60-94f1-412eba22be22","Type":"ContainerStarted","Data":"425e583309c592edb9ae14e1ff9a1553d44c43f55f472d6034fdfddd47d7fed8"} Nov 02 20:43:51.364583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:51.364525 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" podStartSLOduration=2.3645092930000002 podStartE2EDuration="2.364509293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:51.328543381 +0000 UTC m=+3329.845315960" watchObservedRunningTime="2026-11-02 20:43:51.364509293 +0000 UTC m=+3329.881281878" Nov 02 20:43:51.648939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:51.648907 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:51.655442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:51.655413 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:52.179447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:52.179409 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:52.180456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:43:52.180435 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:43:52.696442 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:43:52.696404 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:44:11.797987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.797897 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj"] Nov 02 20:44:11.802191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.802169 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.856475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.856435 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj"] Nov 02 20:44:11.886741 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.886694 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.886948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.886768 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6rz\" (UniqueName: \"kubernetes.io/projected/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-kube-api-access-ss6rz\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.886948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.886808 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.897668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.897640 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:44:11.897668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.897660 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:44:11.897896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.897647 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:44:11.897896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.897641 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:44:11.987530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.987495 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6rz\" (UniqueName: \"kubernetes.io/projected/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-kube-api-access-ss6rz\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.987707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.987543 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.987707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.987679 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.988251 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.988228 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:11.990042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:11.990006 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:12.040593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:12.040556 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6rz\" (UniqueName: \"kubernetes.io/projected/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-kube-api-access-ss6rz\") pod \"maas-api-revoke-sub-af1a238e-ndpkj\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:12.127882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:12.127803 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:12.507611 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:44:12.507582 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213a09d0_3c6c_4d8f_ba46_c5e430482dd0.slice/crio-786f686be001e087cdf9442e973ec97cd40ebe897384db213555a57453b9d685 WatchSource:0}: Error finding container 786f686be001e087cdf9442e973ec97cd40ebe897384db213555a57453b9d685: Status 404 returned error can't find the container with id 786f686be001e087cdf9442e973ec97cd40ebe897384db213555a57453b9d685 Nov 02 20:44:12.508296 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:12.508250 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj"] Nov 02 20:44:13.343264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:13.343235 5800 generic.go:356] "Generic (PLEG): container finished" podID="213a09d0-3c6c-4d8f-ba46-c5e430482dd0" containerID="44098c61bf1af9fb4253ec1d34ce22852392927afd69e876c6788635badd10a3" exitCode=0 Nov 02 20:44:13.343732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:13.343331 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" event={"ID":"213a09d0-3c6c-4d8f-ba46-c5e430482dd0","Type":"ContainerDied","Data":"44098c61bf1af9fb4253ec1d34ce22852392927afd69e876c6788635badd10a3"} Nov 02 20:44:13.343827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:13.343762 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" event={"ID":"213a09d0-3c6c-4d8f-ba46-c5e430482dd0","Type":"ContainerStarted","Data":"786f686be001e087cdf9442e973ec97cd40ebe897384db213555a57453b9d685"} Nov 02 20:44:14.519412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.519382 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:14.622505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.622428 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-kube-api-access-ss6rz\" (UniqueName: \"kubernetes.io/projected/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-kube-api-access-ss6rz\") pod \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " Nov 02 20:44:14.622505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.622487 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-xks-ca\") pod \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " Nov 02 20:44:14.622706 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.622511 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-openshift-ca\") pod \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\" (UID: \"213a09d0-3c6c-4d8f-ba46-c5e430482dd0\") " Nov 02 20:44:14.622961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.622934 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-openshift-ca" pod "213a09d0-3c6c-4d8f-ba46-c5e430482dd0" (UID: "213a09d0-3c6c-4d8f-ba46-c5e430482dd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:14.630467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.630425 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-xks-ca" pod "213a09d0-3c6c-4d8f-ba46-c5e430482dd0" (UID: "213a09d0-3c6c-4d8f-ba46-c5e430482dd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:14.630467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.630457 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-kube-api-access-ss6rz" pod "213a09d0-3c6c-4d8f-ba46-c5e430482dd0" (UID: "213a09d0-3c6c-4d8f-ba46-c5e430482dd0"). InnerVolumeSpecName "kube-api-access-ss6rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:14.723922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.723877 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss6rz\" (UniqueName: \"kubernetes.io/projected/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-kube-api-access-ss6rz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.723922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.723908 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.723922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:14.723919 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/213a09d0-3c6c-4d8f-ba46-c5e430482dd0-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:15.360832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:15.360799 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" Nov 02 20:44:15.360832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:15.360822 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj" event={"ID":"213a09d0-3c6c-4d8f-ba46-c5e430482dd0","Type":"ContainerDied","Data":"786f686be001e087cdf9442e973ec97cd40ebe897384db213555a57453b9d685"} Nov 02 20:44:15.361057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:15.360870 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="786f686be001e087cdf9442e973ec97cd40ebe897384db213555a57453b9d685" Nov 02 20:44:21.203003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:21.202954 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl"] Nov 02 20:44:21.203692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:21.203299 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" containerID="cri-o://de1279d8d67bb7c115af1a5ca9650fbcae5badea73fb8dd1f85a9d9641189260" gracePeriod=30 Nov 02 20:44:22.688362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.688324 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd"] Nov 02 20:44:22.689343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.689322 5800 cpu_manager.go:414] "containerMap: removing container" podUID="213a09d0-3c6c-4d8f-ba46-c5e430482dd0" containerName="revoke-keys" Nov 02 20:44:22.689432 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.689344 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213a09d0-3c6c-4d8f-ba46-c5e430482dd0" containerName="revoke-keys" Nov 02 20:44:22.689498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.689487 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="213a09d0-3c6c-4d8f-ba46-c5e430482dd0" containerName="revoke-keys" Nov 02 20:44:22.693496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.693473 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.699299 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:44:22.699260 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:44:22.742155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.742123 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd"] Nov 02 20:44:22.784387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.784354 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:44:22.784640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.784379 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:44:22.784640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.784420 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:44:22.788496 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.788473 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:44:22.821107 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.821066 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.821571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.821138 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2glk\" (UniqueName: \"kubernetes.io/projected/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-kube-api-access-n2glk\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.821571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.821249 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.922631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.922590 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.922839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.922657 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2glk\" (UniqueName: \"kubernetes.io/projected/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-kube-api-access-n2glk\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.922839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.922737 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.923524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.923500 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.925077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.925059 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:22.975323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:22.975225 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2glk\" (UniqueName: \"kubernetes.io/projected/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-kube-api-access-n2glk\") pod \"maas-api-revoke-sub-eb799601-tdpmd\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:23.008481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:23.008448 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:23.379537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:23.379506 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd"] Nov 02 20:44:23.379701 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:44:23.379679 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd59fbce_d0e5_4ec3_aaad_356fa44ea041.slice/crio-4b275ce080eb84bccb105c94eaa210b0e4883adc19f3ff231cb29b36a8c1a81e WatchSource:0}: Error finding container 4b275ce080eb84bccb105c94eaa210b0e4883adc19f3ff231cb29b36a8c1a81e: Status 404 returned error can't find the container with id 4b275ce080eb84bccb105c94eaa210b0e4883adc19f3ff231cb29b36a8c1a81e Nov 02 20:44:23.421826 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:23.421789 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" event={"ID":"fd59fbce-d0e5-4ec3-aaad-356fa44ea041","Type":"ContainerStarted","Data":"4b275ce080eb84bccb105c94eaa210b0e4883adc19f3ff231cb29b36a8c1a81e"} Nov 02 20:44:24.431118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:24.431078 5800 generic.go:356] "Generic (PLEG): container finished" podID="fd59fbce-d0e5-4ec3-aaad-356fa44ea041" containerID="903fbdb625c03f8c278dd758e23c6f9dc8818b00d031d6fe07c040cc2420e582" exitCode=0 Nov 02 20:44:24.431638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:24.431170 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" event={"ID":"fd59fbce-d0e5-4ec3-aaad-356fa44ea041","Type":"ContainerDied","Data":"903fbdb625c03f8c278dd758e23c6f9dc8818b00d031d6fe07c040cc2420e582"} Nov 02 20:44:24.574702 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:24.574667 5800 patch_prober.go:29] interesting pod/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:44:24.574888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:24.574721 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:44:25.610609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.610575 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:25.757722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.757688 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-kube-api-access-n2glk\" (UniqueName: \"kubernetes.io/projected/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-kube-api-access-n2glk\") pod \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " Nov 02 20:44:25.757937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.757836 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-openshift-ca\") pod \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " Nov 02 20:44:25.758033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.758012 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-xks-ca\") pod \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\" (UID: \"fd59fbce-d0e5-4ec3-aaad-356fa44ea041\") " Nov 02 20:44:25.758268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.758236 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-openshift-ca" pod "fd59fbce-d0e5-4ec3-aaad-356fa44ea041" (UID: "fd59fbce-d0e5-4ec3-aaad-356fa44ea041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:25.758566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.758548 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:25.760712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.760687 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-xks-ca" pod "fd59fbce-d0e5-4ec3-aaad-356fa44ea041" (UID: "fd59fbce-d0e5-4ec3-aaad-356fa44ea041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:25.762449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.762414 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-kube-api-access-n2glk" pod "fd59fbce-d0e5-4ec3-aaad-356fa44ea041" (UID: "fd59fbce-d0e5-4ec3-aaad-356fa44ea041"). InnerVolumeSpecName "kube-api-access-n2glk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:25.859598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.859562 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:25.859598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:25.859589 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2glk\" (UniqueName: \"kubernetes.io/projected/fd59fbce-d0e5-4ec3-aaad-356fa44ea041-kube-api-access-n2glk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.447883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.447851 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" Nov 02 20:44:26.447883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.447873 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd" event={"ID":"fd59fbce-d0e5-4ec3-aaad-356fa44ea041","Type":"ContainerDied","Data":"4b275ce080eb84bccb105c94eaa210b0e4883adc19f3ff231cb29b36a8c1a81e"} Nov 02 20:44:26.448159 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.447927 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b275ce080eb84bccb105c94eaa210b0e4883adc19f3ff231cb29b36a8c1a81e" Nov 02 20:44:26.450836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.450802 5800 generic.go:356] "Generic (PLEG): container finished" podID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerID="de1279d8d67bb7c115af1a5ca9650fbcae5badea73fb8dd1f85a9d9641189260" exitCode=0 Nov 02 20:44:26.451010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.450896 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" event={"ID":"ffaa138c-489b-4b82-99fa-329c35ddecc7","Type":"ContainerDied","Data":"de1279d8d67bb7c115af1a5ca9650fbcae5badea73fb8dd1f85a9d9641189260"} Nov 02 20:44:26.469026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.469002 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:44:26.567560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.567520 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-podinfo\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.567757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.567668 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-credential-socket\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.567757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.567728 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffaa138c-489b-4b82-99fa-329c35ddecc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa138c-489b-4b82-99fa-329c35ddecc7-istiod-ca-cert\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.567871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.567816 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-socket\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.567928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.567907 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-data\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.568015 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.567990 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-certs\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.568079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568014 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-credential-socket" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:26.568079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568033 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-kube-api-access-qzjhg\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-kube-api-access-qzjhg\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.568187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568107 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-socket" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:26.568187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568134 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-token\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.568187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568169 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-envoy\") pod \"ffaa138c-489b-4b82-99fa-329c35ddecc7\" (UID: \"ffaa138c-489b-4b82-99fa-329c35ddecc7\") " Nov 02 20:44:26.568378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568188 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffaa138c-489b-4b82-99fa-329c35ddecc7-istiod-ca-cert" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:26.568378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568217 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-data" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:26.568378 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568330 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-certs" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:26.568760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568734 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.568847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568770 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.568847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568787 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffaa138c-489b-4b82-99fa-329c35ddecc7-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.568847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568805 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.568847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.568818 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.569984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.569959 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-podinfo" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:26.570632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.570610 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-token" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:26.570721 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.570692 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-kube-api-access-qzjhg" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "kube-api-access-qzjhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:26.570890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.570872 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-envoy" pod "ffaa138c-489b-4b82-99fa-329c35ddecc7" (UID: "ffaa138c-489b-4b82-99fa-329c35ddecc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:26.669815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.669779 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjhg\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-kube-api-access-qzjhg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.669815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.669811 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.669815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.669821 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:26.669815 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:26.669831 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffaa138c-489b-4b82-99fa-329c35ddecc7-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:27.459170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:27.459133 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" event={"ID":"ffaa138c-489b-4b82-99fa-329c35ddecc7","Type":"ContainerDied","Data":"fff5190488fd2b6f3f243344677e13180a3acf2cb026598f78c6f8532b32f159"} Nov 02 20:44:27.459374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:27.459195 5800 scope.go:122] "RemoveContainer" containerID="de1279d8d67bb7c115af1a5ca9650fbcae5badea73fb8dd1f85a9d9641189260" Nov 02 20:44:27.459374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:27.459151 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl" Nov 02 20:44:27.613589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:27.613556 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl"] Nov 02 20:44:27.660777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:27.660747 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-14d090-openshift-default-6b8f664c4f-lrnfl"] Nov 02 20:44:29.623065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:29.623032 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" path="/var/lib/kubelet/pods/ffaa138c-489b-4b82-99fa-329c35ddecc7/volumes" Nov 02 20:44:44.477732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:44.477696 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j"] Nov 02 20:44:44.478230 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:44.477943 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" podUID="2bb290aa-6a87-4d60-94f1-412eba22be22" containerName="istio-proxy" containerID="cri-o://425e583309c592edb9ae14e1ff9a1553d44c43f55f472d6034fdfddd47d7fed8" gracePeriod=30 Nov 02 20:44:49.641001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.640957 5800 generic.go:356] "Generic (PLEG): container finished" podID="2bb290aa-6a87-4d60-94f1-412eba22be22" containerID="425e583309c592edb9ae14e1ff9a1553d44c43f55f472d6034fdfddd47d7fed8" exitCode=0 Nov 02 20:44:49.641406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.641029 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" event={"ID":"2bb290aa-6a87-4d60-94f1-412eba22be22","Type":"ContainerDied","Data":"425e583309c592edb9ae14e1ff9a1553d44c43f55f472d6034fdfddd47d7fed8"} Nov 02 20:44:49.757714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.757231 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:44:49.856087 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856050 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-certs\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856087 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856092 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-socket\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856193 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-kube-api-access-5h4tj\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-kube-api-access-5h4tj\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856240 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-token\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856331 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-credential-socket\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856366 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb290aa-6a87-4d60-94f1-412eba22be22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb290aa-6a87-4d60-94f1-412eba22be22-istiod-ca-cert\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856415 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-data\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856417 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-certs" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:49.856630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856475 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-envoy\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856630 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856519 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-podinfo\") pod \"2bb290aa-6a87-4d60-94f1-412eba22be22\" (UID: \"2bb290aa-6a87-4d60-94f1-412eba22be22\") " Nov 02 20:44:49.856868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856514 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-socket" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:49.856936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856650 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-credential-socket" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:49.856992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856698 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-data" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:49.857114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.857093 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.857187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.857120 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.857187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.857134 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.857187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.857148 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.857187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.856781 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb290aa-6a87-4d60-94f1-412eba22be22-istiod-ca-cert" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:49.859009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.858980 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-kube-api-access-5h4tj" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "kube-api-access-5h4tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:49.859124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.859087 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-podinfo" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:49.859176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.859131 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-envoy" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:49.859224 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.859205 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-token" pod "2bb290aa-6a87-4d60-94f1-412eba22be22" (UID: "2bb290aa-6a87-4d60-94f1-412eba22be22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:49.957884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.957846 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.957884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.957877 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.957884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.957888 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h4tj\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-kube-api-access-5h4tj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.958131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.957898 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb290aa-6a87-4d60-94f1-412eba22be22-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:49.958131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:49.957907 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb290aa-6a87-4d60-94f1-412eba22be22-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.650112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:50.650076 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" Nov 02 20:44:50.650112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:50.650093 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j" event={"ID":"2bb290aa-6a87-4d60-94f1-412eba22be22","Type":"ContainerDied","Data":"af55a5b7fa93476ac7297a872e77756034aa05938a6781a0015a7d82bb6c5d47"} Nov 02 20:44:50.650698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:50.650178 5800 scope.go:122] "RemoveContainer" containerID="425e583309c592edb9ae14e1ff9a1553d44c43f55f472d6034fdfddd47d7fed8" Nov 02 20:44:50.804968 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:50.804931 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j"] Nov 02 20:44:50.809510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:50.809479 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-09e014-openshift-default-5ccc6ffc47-56n8j"] Nov 02 20:44:51.624898 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:44:51.624865 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb290aa-6a87-4d60-94f1-412eba22be22" path="/var/lib/kubelet/pods/2bb290aa-6a87-4d60-94f1-412eba22be22/volumes" Nov 02 20:44:52.697704 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:44:52.697666 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:45:00.164602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.164569 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj"] Nov 02 20:45:00.165441 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165418 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2bb290aa-6a87-4d60-94f1-412eba22be22" containerName="istio-proxy" Nov 02 20:45:00.165568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165442 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb290aa-6a87-4d60-94f1-412eba22be22" containerName="istio-proxy" Nov 02 20:45:00.165568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165485 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" Nov 02 20:45:00.165568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165495 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" Nov 02 20:45:00.165568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165509 5800 cpu_manager.go:414] "containerMap: removing container" podUID="fd59fbce-d0e5-4ec3-aaad-356fa44ea041" containerName="revoke-keys" Nov 02 20:45:00.165568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165517 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd59fbce-d0e5-4ec3-aaad-356fa44ea041" containerName="revoke-keys" Nov 02 20:45:00.165837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165689 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd59fbce-d0e5-4ec3-aaad-356fa44ea041" containerName="revoke-keys" Nov 02 20:45:00.165837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165716 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb290aa-6a87-4d60-94f1-412eba22be22" containerName="istio-proxy" Nov 02 20:45:00.165837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.165731 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffaa138c-489b-4b82-99fa-329c35ddecc7" containerName="istio-proxy" Nov 02 20:45:00.169805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.169776 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8"] Nov 02 20:45:00.169956 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.169919 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.174912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.174887 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd"] Nov 02 20:45:00.175044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.175018 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.180201 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.180183 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.211273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.211245 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz"] Nov 02 20:45:00.215857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.215834 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq"] Nov 02 20:45:00.216014 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.215993 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.220751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.220729 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8"] Nov 02 20:45:00.220875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.220859 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.225334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.225310 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw"] Nov 02 20:45:00.225461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.225385 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.244261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.244232 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.254568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.253091 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh"] Nov 02 20:45:00.255373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.255349 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:45:00.255550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.255432 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:45:00.257063 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.257039 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:45:00.262162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.262127 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r"] Nov 02 20:45:00.262389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.262370 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.267755 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.267731 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv"] Nov 02 20:45:00.267954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.267935 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.273376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.273353 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8"] Nov 02 20:45:00.273495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.273396 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz"] Nov 02 20:45:00.273495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.273420 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj"] Nov 02 20:45:00.273495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.273454 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.287949 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.287925 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrgrv\" (UniqueName: \"kubernetes.io/projected/609dc9c2-64ec-40b0-b150-6af0e275ecc5-kube-api-access-jrgrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.288066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.287960 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxnnz\" (UniqueName: \"kubernetes.io/projected/8ca47706-365c-42ab-b9b1-fc865aa53f6f-kube-api-access-fxnnz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.288118 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.288087 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ffa52-e457-4879-bcac-7cfcf926e29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.288218 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.288187 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca47706-365c-42ab-b9b1-fc865aa53f6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.288292 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.288236 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5drm\" (UniqueName: \"kubernetes.io/projected/b42ffa52-e457-4879-bcac-7cfcf926e29f-kube-api-access-n5drm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.288292 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.288265 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/609dc9c2-64ec-40b0-b150-6af0e275ecc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.302878 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.302843 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd"] Nov 02 20:45:00.307967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.307933 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq"] Nov 02 20:45:00.315888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.313876 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8"] Nov 02 20:45:00.318194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.318171 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh"] Nov 02 20:45:00.352328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.351974 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r"] Nov 02 20:45:00.352328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.352052 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv"] Nov 02 20:45:00.368853 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.367328 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw"] Nov 02 20:45:00.389403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389365 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxnnz\" (UniqueName: \"kubernetes.io/projected/8ca47706-365c-42ab-b9b1-fc865aa53f6f-kube-api-access-fxnnz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.389603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389413 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ffa52-e457-4879-bcac-7cfcf926e29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.389603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389440 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02616515-a0a6-427a-bcc9-101c1860a2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.389603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389462 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f84c0a4-90b0-43b6-91da-b051569c7cc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894205-x5zdv\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.389603 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389506 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qbm\" (UniqueName: \"kubernetes.io/projected/b04d196c-4c5a-47bb-a174-e6c399e6e1de-kube-api-access-t9qbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.389799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389624 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.389799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389713 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvvq\" (UniqueName: \"kubernetes.io/projected/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-kube-api-access-2dvvq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.389799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389760 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca47706-365c-42ab-b9b1-fc865aa53f6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.389799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389788 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04d196c-4c5a-47bb-a174-e6c399e6e1de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.389978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389820 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5drm\" (UniqueName: \"kubernetes.io/projected/b42ffa52-e457-4879-bcac-7cfcf926e29f-kube-api-access-n5drm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.389978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389854 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/609dc9c2-64ec-40b0-b150-6af0e275ecc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.389978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389889 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacd99ac-5892-4452-88f6-d99c87903907-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.389978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.389923 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j756\" (UniqueName: \"kubernetes.io/projected/25ee92b8-8bbe-4513-8973-d48c4eba73df-kube-api-access-5j756\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.390218 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390062 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmb6m\" (UniqueName: \"kubernetes.io/projected/9f84c0a4-90b0-43b6-91da-b051569c7cc2-kube-api-access-bmb6m\") pod \"maas-api-key-cleanup-29894205-x5zdv\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.390218 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390112 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ee92b8-8bbe-4513-8973-d48c4eba73df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.390218 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390154 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xfw\" (UniqueName: \"kubernetes.io/projected/f01c86b0-d4ba-4064-8e09-4538ffd92891-kube-api-access-c9xfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.390416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390254 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74j5l\" (UniqueName: \"kubernetes.io/projected/bacd99ac-5892-4452-88f6-d99c87903907-kube-api-access-74j5l\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.390416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390369 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hww\" (UniqueName: \"kubernetes.io/projected/02616515-a0a6-427a-bcc9-101c1860a2d7-kube-api-access-65hww\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.390535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390436 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01c86b0-d4ba-4064-8e09-4538ffd92891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.390535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390485 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrgrv\" (UniqueName: \"kubernetes.io/projected/609dc9c2-64ec-40b0-b150-6af0e275ecc5-kube-api-access-jrgrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.390647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390620 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/609dc9c2-64ec-40b0-b150-6af0e275ecc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.390707 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390679 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca47706-365c-42ab-b9b1-fc865aa53f6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.390784 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.390734 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ffa52-e457-4879-bcac-7cfcf926e29f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.443174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.443089 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrgrv\" (UniqueName: \"kubernetes.io/projected/609dc9c2-64ec-40b0-b150-6af0e275ecc5-kube-api-access-jrgrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.443368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.443300 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxnnz\" (UniqueName: \"kubernetes.io/projected/8ca47706-365c-42ab-b9b1-fc865aa53f6f-kube-api-access-fxnnz\") pod \"maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.443368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.443308 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5drm\" (UniqueName: \"kubernetes.io/projected/b42ffa52-e457-4879-bcac-7cfcf926e29f-kube-api-access-n5drm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.487554 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.487523 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491606 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02616515-a0a6-427a-bcc9-101c1860a2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491640 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f84c0a4-90b0-43b6-91da-b051569c7cc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894205-x5zdv\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491674 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qbm\" (UniqueName: \"kubernetes.io/projected/b04d196c-4c5a-47bb-a174-e6c399e6e1de-kube-api-access-t9qbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491697 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491805 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvvq\" (UniqueName: \"kubernetes.io/projected/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-kube-api-access-2dvvq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491870 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04d196c-4c5a-47bb-a174-e6c399e6e1de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491917 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacd99ac-5892-4452-88f6-d99c87903907-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.491976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491955 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j756\" (UniqueName: \"kubernetes.io/projected/25ee92b8-8bbe-4513-8973-d48c4eba73df-kube-api-access-5j756\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.492487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.491985 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmb6m\" (UniqueName: \"kubernetes.io/projected/9f84c0a4-90b0-43b6-91da-b051569c7cc2-kube-api-access-bmb6m\") pod \"maas-api-key-cleanup-29894205-x5zdv\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.492487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492024 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ee92b8-8bbe-4513-8973-d48c4eba73df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.492487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492061 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xfw\" (UniqueName: \"kubernetes.io/projected/f01c86b0-d4ba-4064-8e09-4538ffd92891-kube-api-access-c9xfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.492487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492135 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74j5l\" (UniqueName: \"kubernetes.io/projected/bacd99ac-5892-4452-88f6-d99c87903907-kube-api-access-74j5l\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.492487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492172 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hww\" (UniqueName: \"kubernetes.io/projected/02616515-a0a6-427a-bcc9-101c1860a2d7-kube-api-access-65hww\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.492487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492235 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01c86b0-d4ba-4064-8e09-4538ffd92891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.492487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492464 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02616515-a0a6-427a-bcc9-101c1860a2d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.492837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492530 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.492837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492753 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacd99ac-5892-4452-88f6-d99c87903907-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.493009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.492992 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01c86b0-d4ba-4064-8e09-4538ffd92891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.493076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.493054 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04d196c-4c5a-47bb-a174-e6c399e6e1de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.493116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.493062 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f84c0a4-90b0-43b6-91da-b051569c7cc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894205-x5zdv\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.493233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.493217 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ee92b8-8bbe-4513-8973-d48c4eba73df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.498104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.495728 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:00.507085 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.507061 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:00.549027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.548994 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmb6m\" (UniqueName: \"kubernetes.io/projected/9f84c0a4-90b0-43b6-91da-b051569c7cc2-kube-api-access-bmb6m\") pod \"maas-api-key-cleanup-29894205-x5zdv\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.549664 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.549614 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hww\" (UniqueName: \"kubernetes.io/projected/02616515-a0a6-427a-bcc9-101c1860a2d7-kube-api-access-65hww\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.554511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.553785 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j756\" (UniqueName: \"kubernetes.io/projected/25ee92b8-8bbe-4513-8973-d48c4eba73df-kube-api-access-5j756\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.554511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.553979 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvvq\" (UniqueName: \"kubernetes.io/projected/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-kube-api-access-2dvvq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.554511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.554469 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xfw\" (UniqueName: \"kubernetes.io/projected/f01c86b0-d4ba-4064-8e09-4538ffd92891-kube-api-access-c9xfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.558884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.556523 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qbm\" (UniqueName: \"kubernetes.io/projected/b04d196c-4c5a-47bb-a174-e6c399e6e1de-kube-api-access-t9qbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.559131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.559086 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74j5l\" (UniqueName: \"kubernetes.io/projected/bacd99ac-5892-4452-88f6-d99c87903907-kube-api-access-74j5l\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.574744 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.574490 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:00.577974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.577923 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:00.586918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.586890 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:00.596714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.596473 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:00.608046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.607840 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:00.624413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.624227 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:00.839487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.839195 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:00.907119 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.907075 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8"] Nov 02 20:45:00.907558 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:00.907523 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42ffa52_e457_4879_bcac_7cfcf926e29f.slice/crio-c6764f767c7234e0d3d0b456e8f5d22c4835ccfcbb2c5fc6f5c1cbf4a82a953a WatchSource:0}: Error finding container c6764f767c7234e0d3d0b456e8f5d22c4835ccfcbb2c5fc6f5c1cbf4a82a953a: Status 404 returned error can't find the container with id c6764f767c7234e0d3d0b456e8f5d22c4835ccfcbb2c5fc6f5c1cbf4a82a953a Nov 02 20:45:00.956698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.951385 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd"] Nov 02 20:45:00.966175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:00.965613 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj"] Nov 02 20:45:00.973639 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:00.972605 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609dc9c2_64ec_40b0_b150_6af0e275ecc5.slice/crio-9d3c49a3c014bd7903721305f6ad874daa68dcdda9601893aff1245b22e282f7 WatchSource:0}: Error finding container 9d3c49a3c014bd7903721305f6ad874daa68dcdda9601893aff1245b22e282f7: Status 404 returned error can't find the container with id 9d3c49a3c014bd7903721305f6ad874daa68dcdda9601893aff1245b22e282f7 Nov 02 20:45:01.034326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.033933 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq"] Nov 02 20:45:01.073077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.060497 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw"] Nov 02 20:45:01.073077 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:01.071955 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacd99ac_5892_4452_88f6_d99c87903907.slice/crio-d782db716d2d4c9a446493ad1ef1a529e3af68ce2a466d347bd0df5182c03883 WatchSource:0}: Error finding container d782db716d2d4c9a446493ad1ef1a529e3af68ce2a466d347bd0df5182c03883: Status 404 returned error can't find the container with id d782db716d2d4c9a446493ad1ef1a529e3af68ce2a466d347bd0df5182c03883 Nov 02 20:45:01.090252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.090228 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8"] Nov 02 20:45:01.093976 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.093942 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh"] Nov 02 20:45:01.166622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.166584 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv"] Nov 02 20:45:01.167019 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:01.166737 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04d196c_4c5a_47bb_a174_e6c399e6e1de.slice/crio-f0bed840f4d5061d32601cfe47bb67ef48d03e88b2e4c97481a5b56f5ecf881d WatchSource:0}: Error finding container f0bed840f4d5061d32601cfe47bb67ef48d03e88b2e4c97481a5b56f5ecf881d: Status 404 returned error can't find the container with id f0bed840f4d5061d32601cfe47bb67ef48d03e88b2e4c97481a5b56f5ecf881d Nov 02 20:45:01.168652 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:01.168604 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f84c0a4_90b0_43b6_91da_b051569c7cc2.slice/crio-dcc8791982445ebc90e544d9431eb39671c45b381e4939c63da888493085a9e6 WatchSource:0}: Error finding container dcc8791982445ebc90e544d9431eb39671c45b381e4939c63da888493085a9e6: Status 404 returned error can't find the container with id dcc8791982445ebc90e544d9431eb39671c45b381e4939c63da888493085a9e6 Nov 02 20:45:01.171358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.171262 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r"] Nov 02 20:45:01.257655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.257596 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz"] Nov 02 20:45:01.344222 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:01.344181 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ee92b8_8bbe_4513_8973_d48c4eba73df.slice/crio-dc7db7b7fea1b6cb6f74d1fbe2f6402a56421d990735be82eca57dcdf0f9b898 WatchSource:0}: Error finding container dc7db7b7fea1b6cb6f74d1fbe2f6402a56421d990735be82eca57dcdf0f9b898: Status 404 returned error can't find the container with id dc7db7b7fea1b6cb6f74d1fbe2f6402a56421d990735be82eca57dcdf0f9b898 Nov 02 20:45:01.770351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.770273 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" event={"ID":"8ca47706-365c-42ab-b9b1-fc865aa53f6f","Type":"ContainerStarted","Data":"19925f46c846d71cd87312e79f6d20fd74debca057280054b7cedad4d67b393a"} Nov 02 20:45:01.770521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.770370 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" event={"ID":"8ca47706-365c-42ab-b9b1-fc865aa53f6f","Type":"ContainerStarted","Data":"0e80d38ee42ca6b04495757d0adb2cd04938ad02855a8db052c99e025a903f0d"} Nov 02 20:45:01.773520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.772935 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" event={"ID":"bacd99ac-5892-4452-88f6-d99c87903907","Type":"ContainerStarted","Data":"d782db716d2d4c9a446493ad1ef1a529e3af68ce2a466d347bd0df5182c03883"} Nov 02 20:45:01.774746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.774659 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" event={"ID":"a697bc87-eecd-44b6-80f4-fcb0f95ae47e","Type":"ContainerStarted","Data":"17e8c98245387fe6cb7946bb0732d0ef7e62a0e7790e833d6017eab518bbd603"} Nov 02 20:45:01.774746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.774715 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" event={"ID":"a697bc87-eecd-44b6-80f4-fcb0f95ae47e","Type":"ContainerStarted","Data":"a66d1551a3dd9f0e04eb6852dc9fe85e32bf5aa9f8d95b51c4b3ded66abdd4f3"} Nov 02 20:45:01.776649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.776620 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" event={"ID":"b04d196c-4c5a-47bb-a174-e6c399e6e1de","Type":"ContainerStarted","Data":"e815e1041dae2c1b23b7db6c4fd4dcbcb2c925e38a34eff4140f69dd38dc8cda"} Nov 02 20:45:01.776764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.776673 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" event={"ID":"b04d196c-4c5a-47bb-a174-e6c399e6e1de","Type":"ContainerStarted","Data":"f0bed840f4d5061d32601cfe47bb67ef48d03e88b2e4c97481a5b56f5ecf881d"} Nov 02 20:45:01.778820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.778789 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" event={"ID":"609dc9c2-64ec-40b0-b150-6af0e275ecc5","Type":"ContainerStarted","Data":"db796a17bcdedd08f0b4005983424afab669775eb0cafdfeb76b2eb3bb2846a1"} Nov 02 20:45:01.778938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.778842 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" event={"ID":"609dc9c2-64ec-40b0-b150-6af0e275ecc5","Type":"ContainerStarted","Data":"9d3c49a3c014bd7903721305f6ad874daa68dcdda9601893aff1245b22e282f7"} Nov 02 20:45:01.780717 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.780686 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" event={"ID":"b42ffa52-e457-4879-bcac-7cfcf926e29f","Type":"ContainerStarted","Data":"67a304b7725adf70f36dbeee3f92cc755fb5b192997c768b9db2de60547c95bb"} Nov 02 20:45:01.780840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.780739 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" event={"ID":"b42ffa52-e457-4879-bcac-7cfcf926e29f","Type":"ContainerStarted","Data":"c6764f767c7234e0d3d0b456e8f5d22c4835ccfcbb2c5fc6f5c1cbf4a82a953a"} Nov 02 20:45:01.782495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.782470 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" event={"ID":"25ee92b8-8bbe-4513-8973-d48c4eba73df","Type":"ContainerStarted","Data":"dc7db7b7fea1b6cb6f74d1fbe2f6402a56421d990735be82eca57dcdf0f9b898"} Nov 02 20:45:01.784211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.784190 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" event={"ID":"9f84c0a4-90b0-43b6-91da-b051569c7cc2","Type":"ContainerStarted","Data":"95af79486df0b7edbe308a82e03b5c4ed81bf5e035f21a19c2ae1d3adc59696d"} Nov 02 20:45:01.784359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.784232 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" event={"ID":"9f84c0a4-90b0-43b6-91da-b051569c7cc2","Type":"ContainerStarted","Data":"dcc8791982445ebc90e544d9431eb39671c45b381e4939c63da888493085a9e6"} Nov 02 20:45:01.786075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.786042 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" event={"ID":"f01c86b0-d4ba-4064-8e09-4538ffd92891","Type":"ContainerStarted","Data":"6b63da7a897ce8edf18108d6b8105034b4afa3514d99f008719817ab87b9692c"} Nov 02 20:45:01.786181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.786099 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" event={"ID":"f01c86b0-d4ba-4064-8e09-4538ffd92891","Type":"ContainerStarted","Data":"5bc95ee982ee10693c95870109d9d739f9a00c2803ecfd737f12629c4bc3cceb"} Nov 02 20:45:01.788225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.788160 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" event={"ID":"02616515-a0a6-427a-bcc9-101c1860a2d7","Type":"ContainerStarted","Data":"343cd87970fd8e4a3b576095405a84b6125f661d62bc9406cd1544361ce3e965"} Nov 02 20:45:01.788225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.788196 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" event={"ID":"02616515-a0a6-427a-bcc9-101c1860a2d7","Type":"ContainerStarted","Data":"6f17176d4e473af59251b2d78de35814510f5eef7576668b25cf49ea93cccefb"} Nov 02 20:45:01.882742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.882659 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" podStartSLOduration=1.882635748 podStartE2EDuration="1.882635748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:01.877860961 +0000 UTC m=+3400.394633547" watchObservedRunningTime="2026-11-02 20:45:01.882635748 +0000 UTC m=+3400.399408333" Nov 02 20:45:01.988844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:01.988772 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" podStartSLOduration=1.988751523 podStartE2EDuration="1.988751523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:01.985791645 +0000 UTC m=+3400.502564231" watchObservedRunningTime="2026-11-02 20:45:01.988751523 +0000 UTC m=+3400.505524110" Nov 02 20:45:02.094489 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.094418 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" podStartSLOduration=2.094398008 podStartE2EDuration="2.094398008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.092314971 +0000 UTC m=+3400.609087559" watchObservedRunningTime="2026-11-02 20:45:02.094398008 +0000 UTC m=+3400.611170594" Nov 02 20:45:02.202210 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.202149 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" podStartSLOduration=2.202130144 podStartE2EDuration="2.202130144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.19753497 +0000 UTC m=+3400.714307556" watchObservedRunningTime="2026-11-02 20:45:02.202130144 +0000 UTC m=+3400.718902728" Nov 02 20:45:02.307535 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.307410 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" podStartSLOduration=2.307388822 podStartE2EDuration="2.307388822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.303477603 +0000 UTC m=+3400.820250190" watchObservedRunningTime="2026-11-02 20:45:02.307388822 +0000 UTC m=+3400.824161411" Nov 02 20:45:02.412024 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.411966 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" podStartSLOduration=2.411953039 podStartE2EDuration="2.411953039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.408976063 +0000 UTC m=+3400.925748650" watchObservedRunningTime="2026-11-02 20:45:02.411953039 +0000 UTC m=+3400.928725625" Nov 02 20:45:02.520292 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.520226 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" podStartSLOduration=2.520210063 podStartE2EDuration="2.520210063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.51492583 +0000 UTC m=+3401.031698416" watchObservedRunningTime="2026-11-02 20:45:02.520210063 +0000 UTC m=+3401.036982702" Nov 02 20:45:02.635809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.635755 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" podStartSLOduration=2.635740916 podStartE2EDuration="2.635740916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.633729308 +0000 UTC m=+3401.150501896" watchObservedRunningTime="2026-11-02 20:45:02.635740916 +0000 UTC m=+3401.152513500" Nov 02 20:45:02.744960 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.744903 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" podStartSLOduration=2.7448889899999998 podStartE2EDuration="2.74488899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.741154281 +0000 UTC m=+3401.257926867" watchObservedRunningTime="2026-11-02 20:45:02.74488899 +0000 UTC m=+3401.261661572" Nov 02 20:45:02.800295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.800254 5800 generic.go:356] "Generic (PLEG): container finished" podID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerID="e815e1041dae2c1b23b7db6c4fd4dcbcb2c925e38a34eff4140f69dd38dc8cda" exitCode=60 Nov 02 20:45:02.800486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.800344 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" event={"ID":"b04d196c-4c5a-47bb-a174-e6c399e6e1de","Type":"ContainerDied","Data":"e815e1041dae2c1b23b7db6c4fd4dcbcb2c925e38a34eff4140f69dd38dc8cda"} Nov 02 20:45:02.800779 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.800747 5800 scope.go:122] "RemoveContainer" containerID="e815e1041dae2c1b23b7db6c4fd4dcbcb2c925e38a34eff4140f69dd38dc8cda" Nov 02 20:45:02.802382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.802360 5800 generic.go:356] "Generic (PLEG): container finished" podID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerID="db796a17bcdedd08f0b4005983424afab669775eb0cafdfeb76b2eb3bb2846a1" exitCode=60 Nov 02 20:45:02.802503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.802445 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" event={"ID":"609dc9c2-64ec-40b0-b150-6af0e275ecc5","Type":"ContainerDied","Data":"db796a17bcdedd08f0b4005983424afab669775eb0cafdfeb76b2eb3bb2846a1"} Nov 02 20:45:02.802673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.802657 5800 scope.go:122] "RemoveContainer" containerID="db796a17bcdedd08f0b4005983424afab669775eb0cafdfeb76b2eb3bb2846a1" Nov 02 20:45:02.804223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.804200 5800 generic.go:356] "Generic (PLEG): container finished" podID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerID="67a304b7725adf70f36dbeee3f92cc755fb5b192997c768b9db2de60547c95bb" exitCode=60 Nov 02 20:45:02.804362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.804294 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" event={"ID":"b42ffa52-e457-4879-bcac-7cfcf926e29f","Type":"ContainerDied","Data":"67a304b7725adf70f36dbeee3f92cc755fb5b192997c768b9db2de60547c95bb"} Nov 02 20:45:02.804600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.804548 5800 scope.go:122] "RemoveContainer" containerID="67a304b7725adf70f36dbeee3f92cc755fb5b192997c768b9db2de60547c95bb" Nov 02 20:45:02.806460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.806441 5800 generic.go:356] "Generic (PLEG): container finished" podID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerID="dbcea7fcc2447c5c107f0b29a33813d7b0bf614f84422f0181c0e65f612c9a31" exitCode=60 Nov 02 20:45:02.806558 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.806524 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" event={"ID":"25ee92b8-8bbe-4513-8973-d48c4eba73df","Type":"ContainerDied","Data":"dbcea7fcc2447c5c107f0b29a33813d7b0bf614f84422f0181c0e65f612c9a31"} Nov 02 20:45:02.806751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.806735 5800 scope.go:122] "RemoveContainer" containerID="dbcea7fcc2447c5c107f0b29a33813d7b0bf614f84422f0181c0e65f612c9a31" Nov 02 20:45:02.807979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.807961 5800 generic.go:356] "Generic (PLEG): container finished" podID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerID="95af79486df0b7edbe308a82e03b5c4ed81bf5e035f21a19c2ae1d3adc59696d" exitCode=60 Nov 02 20:45:02.808065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.808011 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" event={"ID":"9f84c0a4-90b0-43b6-91da-b051569c7cc2","Type":"ContainerDied","Data":"95af79486df0b7edbe308a82e03b5c4ed81bf5e035f21a19c2ae1d3adc59696d"} Nov 02 20:45:02.808250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.808233 5800 scope.go:122] "RemoveContainer" containerID="95af79486df0b7edbe308a82e03b5c4ed81bf5e035f21a19c2ae1d3adc59696d" Nov 02 20:45:02.809745 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.809725 5800 generic.go:356] "Generic (PLEG): container finished" podID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerID="6b63da7a897ce8edf18108d6b8105034b4afa3514d99f008719817ab87b9692c" exitCode=60 Nov 02 20:45:02.809843 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.809818 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" event={"ID":"f01c86b0-d4ba-4064-8e09-4538ffd92891","Type":"ContainerDied","Data":"6b63da7a897ce8edf18108d6b8105034b4afa3514d99f008719817ab87b9692c"} Nov 02 20:45:02.810009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.809993 5800 scope.go:122] "RemoveContainer" containerID="6b63da7a897ce8edf18108d6b8105034b4afa3514d99f008719817ab87b9692c" Nov 02 20:45:02.811562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.811541 5800 generic.go:356] "Generic (PLEG): container finished" podID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerID="343cd87970fd8e4a3b576095405a84b6125f661d62bc9406cd1544361ce3e965" exitCode=60 Nov 02 20:45:02.811672 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.811577 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" event={"ID":"02616515-a0a6-427a-bcc9-101c1860a2d7","Type":"ContainerDied","Data":"343cd87970fd8e4a3b576095405a84b6125f661d62bc9406cd1544361ce3e965"} Nov 02 20:45:02.811801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.811787 5800 scope.go:122] "RemoveContainer" containerID="343cd87970fd8e4a3b576095405a84b6125f661d62bc9406cd1544361ce3e965" Nov 02 20:45:02.814172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.814037 5800 generic.go:356] "Generic (PLEG): container finished" podID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerID="19925f46c846d71cd87312e79f6d20fd74debca057280054b7cedad4d67b393a" exitCode=60 Nov 02 20:45:02.814172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.814130 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" event={"ID":"8ca47706-365c-42ab-b9b1-fc865aa53f6f","Type":"ContainerDied","Data":"19925f46c846d71cd87312e79f6d20fd74debca057280054b7cedad4d67b393a"} Nov 02 20:45:02.814526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.814360 5800 scope.go:122] "RemoveContainer" containerID="19925f46c846d71cd87312e79f6d20fd74debca057280054b7cedad4d67b393a" Nov 02 20:45:02.815807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.815788 5800 generic.go:356] "Generic (PLEG): container finished" podID="bacd99ac-5892-4452-88f6-d99c87903907" containerID="2ae6280e65057537dbbdfe3a75588e50cb5f9b93e991663586b58e5b9a8a6a02" exitCode=60 Nov 02 20:45:02.815895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.815868 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" event={"ID":"bacd99ac-5892-4452-88f6-d99c87903907","Type":"ContainerDied","Data":"2ae6280e65057537dbbdfe3a75588e50cb5f9b93e991663586b58e5b9a8a6a02"} Nov 02 20:45:02.816135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.816117 5800 scope.go:122] "RemoveContainer" containerID="2ae6280e65057537dbbdfe3a75588e50cb5f9b93e991663586b58e5b9a8a6a02" Nov 02 20:45:02.817692 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.817669 5800 generic.go:356] "Generic (PLEG): container finished" podID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerID="17e8c98245387fe6cb7946bb0732d0ef7e62a0e7790e833d6017eab518bbd603" exitCode=60 Nov 02 20:45:02.817757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.817723 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" event={"ID":"a697bc87-eecd-44b6-80f4-fcb0f95ae47e","Type":"ContainerDied","Data":"17e8c98245387fe6cb7946bb0732d0ef7e62a0e7790e833d6017eab518bbd603"} Nov 02 20:45:02.818229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.818168 5800 scope.go:122] "RemoveContainer" containerID="17e8c98245387fe6cb7946bb0732d0ef7e62a0e7790e833d6017eab518bbd603" Nov 02 20:45:02.972846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:02.972787 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" podStartSLOduration=2.972773771 podStartE2EDuration="2.972773771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:02.864429268 +0000 UTC m=+3401.381201856" watchObservedRunningTime="2026-11-02 20:45:02.972773771 +0000 UTC m=+3401.489546403" Nov 02 20:45:03.835062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:03.835009 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" event={"ID":"b04d196c-4c5a-47bb-a174-e6c399e6e1de","Type":"ContainerStarted","Data":"8d3a4fc80af7ac23a77e6bf9f949c6576c6b454a73d29692dc1ea7bffa35cc93"} Nov 02 20:45:03.838020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:03.837987 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" event={"ID":"609dc9c2-64ec-40b0-b150-6af0e275ecc5","Type":"ContainerStarted","Data":"3aaadce729d70aa38ae8137b73b2970fec6378c68c1c27ce4fbde96f22092cf9"} Nov 02 20:45:03.840696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:03.840454 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" event={"ID":"b42ffa52-e457-4879-bcac-7cfcf926e29f","Type":"ContainerStarted","Data":"6787d091af580dfda616a7ad1ec6862e438dc59547ddb150545945e34c65a9a1"} Nov 02 20:45:03.842957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:03.842927 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" event={"ID":"25ee92b8-8bbe-4513-8973-d48c4eba73df","Type":"ContainerStarted","Data":"4e5babf930e0cbe82d21a719297acfffcd246fe08fa7f1a377bd806f2db7a774"} Nov 02 20:45:03.845453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:03.845424 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" event={"ID":"9f84c0a4-90b0-43b6-91da-b051569c7cc2","Type":"ContainerStarted","Data":"b3ec26dd885f9682abeddd51b64e76e63b59ce2d8dd4fb633e3e1b38c6ea735c"} Nov 02 20:45:03.848321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:03.848253 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" event={"ID":"02616515-a0a6-427a-bcc9-101c1860a2d7","Type":"ContainerStarted","Data":"bb266567ea581463bd42ab177f709539d27a4ee75929e533d2a924350c4813d3"} Nov 02 20:45:04.865932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.865835 5800 generic.go:356] "Generic (PLEG): container finished" podID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerID="edf511138db652499bc10886ae7a08548c4c409b553c62d74c75fcbf95fbf489" exitCode=60 Nov 02 20:45:04.866436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.865919 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" event={"ID":"f01c86b0-d4ba-4064-8e09-4538ffd92891","Type":"ContainerDied","Data":"edf511138db652499bc10886ae7a08548c4c409b553c62d74c75fcbf95fbf489"} Nov 02 20:45:04.866436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.865983 5800 scope.go:122] "RemoveContainer" containerID="6b63da7a897ce8edf18108d6b8105034b4afa3514d99f008719817ab87b9692c" Nov 02 20:45:04.866436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.866319 5800 scope.go:122] "RemoveContainer" containerID="edf511138db652499bc10886ae7a08548c4c409b553c62d74c75fcbf95fbf489" Nov 02 20:45:04.866731 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.866706 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq_odh-ai-gateway-infra(f01c86b0-d4ba-4064-8e09-4538ffd92891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" Nov 02 20:45:04.868261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.868217 5800 generic.go:356] "Generic (PLEG): container finished" podID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerID="bb266567ea581463bd42ab177f709539d27a4ee75929e533d2a924350c4813d3" exitCode=60 Nov 02 20:45:04.868386 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.868254 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" event={"ID":"02616515-a0a6-427a-bcc9-101c1860a2d7","Type":"ContainerDied","Data":"bb266567ea581463bd42ab177f709539d27a4ee75929e533d2a924350c4813d3"} Nov 02 20:45:04.868619 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.868601 5800 scope.go:122] "RemoveContainer" containerID="bb266567ea581463bd42ab177f709539d27a4ee75929e533d2a924350c4813d3" Nov 02 20:45:04.868930 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.868907 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh_odh-ai-gateway-infra(02616515-a0a6-427a-bcc9-101c1860a2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" Nov 02 20:45:04.870643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.870618 5800 generic.go:356] "Generic (PLEG): container finished" podID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerID="4aeb194aa1eb5b5493c28aecd7a4cdd8d2d9d5a823ca93bd051ad11789cd7e55" exitCode=60 Nov 02 20:45:04.870759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.870701 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" event={"ID":"8ca47706-365c-42ab-b9b1-fc865aa53f6f","Type":"ContainerDied","Data":"4aeb194aa1eb5b5493c28aecd7a4cdd8d2d9d5a823ca93bd051ad11789cd7e55"} Nov 02 20:45:04.871030 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.871007 5800 scope.go:122] "RemoveContainer" containerID="4aeb194aa1eb5b5493c28aecd7a4cdd8d2d9d5a823ca93bd051ad11789cd7e55" Nov 02 20:45:04.871383 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.871346 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd_odh-ai-gateway-infra(8ca47706-365c-42ab-b9b1-fc865aa53f6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" Nov 02 20:45:04.873220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.873194 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" event={"ID":"bacd99ac-5892-4452-88f6-d99c87903907","Type":"ContainerStarted","Data":"565a93594a263985158021085c080e75e62a0ec0626367e2365e543f673520d4"} Nov 02 20:45:04.876235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.876205 5800 generic.go:356] "Generic (PLEG): container finished" podID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerID="735d9677c271480b8e11b4a5673b2bc22017fc3174af5b79d3c49b62e858c8a5" exitCode=60 Nov 02 20:45:04.876396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.876319 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" event={"ID":"a697bc87-eecd-44b6-80f4-fcb0f95ae47e","Type":"ContainerDied","Data":"735d9677c271480b8e11b4a5673b2bc22017fc3174af5b79d3c49b62e858c8a5"} Nov 02 20:45:04.876582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.876564 5800 scope.go:122] "RemoveContainer" containerID="735d9677c271480b8e11b4a5673b2bc22017fc3174af5b79d3c49b62e858c8a5" Nov 02 20:45:04.876880 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.876854 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8_odh-ai-gateway-infra(a697bc87-eecd-44b6-80f4-fcb0f95ae47e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" Nov 02 20:45:04.878638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.878616 5800 generic.go:356] "Generic (PLEG): container finished" podID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerID="8d3a4fc80af7ac23a77e6bf9f949c6576c6b454a73d29692dc1ea7bffa35cc93" exitCode=60 Nov 02 20:45:04.878734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.878707 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" event={"ID":"b04d196c-4c5a-47bb-a174-e6c399e6e1de","Type":"ContainerDied","Data":"8d3a4fc80af7ac23a77e6bf9f949c6576c6b454a73d29692dc1ea7bffa35cc93"} Nov 02 20:45:04.878989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.878967 5800 scope.go:122] "RemoveContainer" containerID="8d3a4fc80af7ac23a77e6bf9f949c6576c6b454a73d29692dc1ea7bffa35cc93" Nov 02 20:45:04.879415 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.879387 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r_odh-ai-gateway-infra(b04d196c-4c5a-47bb-a174-e6c399e6e1de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" Nov 02 20:45:04.881358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.881333 5800 generic.go:356] "Generic (PLEG): container finished" podID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerID="3aaadce729d70aa38ae8137b73b2970fec6378c68c1c27ce4fbde96f22092cf9" exitCode=60 Nov 02 20:45:04.881945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.881424 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" event={"ID":"609dc9c2-64ec-40b0-b150-6af0e275ecc5","Type":"ContainerDied","Data":"3aaadce729d70aa38ae8137b73b2970fec6378c68c1c27ce4fbde96f22092cf9"} Nov 02 20:45:04.881945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.881712 5800 scope.go:122] "RemoveContainer" containerID="3aaadce729d70aa38ae8137b73b2970fec6378c68c1c27ce4fbde96f22092cf9" Nov 02 20:45:04.884636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.884609 5800 scope.go:122] "RemoveContainer" containerID="343cd87970fd8e4a3b576095405a84b6125f661d62bc9406cd1544361ce3e965" Nov 02 20:45:04.895634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.895596 5800 generic.go:356] "Generic (PLEG): container finished" podID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerID="6787d091af580dfda616a7ad1ec6862e438dc59547ddb150545945e34c65a9a1" exitCode=60 Nov 02 20:45:04.895761 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.895712 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" event={"ID":"b42ffa52-e457-4879-bcac-7cfcf926e29f","Type":"ContainerDied","Data":"6787d091af580dfda616a7ad1ec6862e438dc59547ddb150545945e34c65a9a1"} Nov 02 20:45:04.896271 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.896244 5800 scope.go:122] "RemoveContainer" containerID="6787d091af580dfda616a7ad1ec6862e438dc59547ddb150545945e34c65a9a1" Nov 02 20:45:04.896736 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.896708 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8_odh-ai-gateway-infra(b42ffa52-e457-4879-bcac-7cfcf926e29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" Nov 02 20:45:04.896888 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.896862 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj_odh-ai-gateway-infra(609dc9c2-64ec-40b0-b150-6af0e275ecc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" Nov 02 20:45:04.901485 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.901454 5800 generic.go:356] "Generic (PLEG): container finished" podID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerID="4e5babf930e0cbe82d21a719297acfffcd246fe08fa7f1a377bd806f2db7a774" exitCode=60 Nov 02 20:45:04.901602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.901536 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" event={"ID":"25ee92b8-8bbe-4513-8973-d48c4eba73df","Type":"ContainerDied","Data":"4e5babf930e0cbe82d21a719297acfffcd246fe08fa7f1a377bd806f2db7a774"} Nov 02 20:45:04.901793 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.901774 5800 scope.go:122] "RemoveContainer" containerID="4e5babf930e0cbe82d21a719297acfffcd246fe08fa7f1a377bd806f2db7a774" Nov 02 20:45:04.902091 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.902060 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz_odh-ai-gateway-infra(25ee92b8-8bbe-4513-8973-d48c4eba73df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" Nov 02 20:45:04.903759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.903735 5800 generic.go:356] "Generic (PLEG): container finished" podID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerID="b3ec26dd885f9682abeddd51b64e76e63b59ce2d8dd4fb633e3e1b38c6ea735c" exitCode=60 Nov 02 20:45:04.903869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.903787 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" event={"ID":"9f84c0a4-90b0-43b6-91da-b051569c7cc2","Type":"ContainerDied","Data":"b3ec26dd885f9682abeddd51b64e76e63b59ce2d8dd4fb633e3e1b38c6ea735c"} Nov 02 20:45:04.904064 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.904047 5800 scope.go:122] "RemoveContainer" containerID="b3ec26dd885f9682abeddd51b64e76e63b59ce2d8dd4fb633e3e1b38c6ea735c" Nov 02 20:45:04.904348 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:04.904327 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894205-x5zdv_odh-ai-gateway-infra(9f84c0a4-90b0-43b6-91da-b051569c7cc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" Nov 02 20:45:04.910757 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.910737 5800 scope.go:122] "RemoveContainer" containerID="19925f46c846d71cd87312e79f6d20fd74debca057280054b7cedad4d67b393a" Nov 02 20:45:04.927254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.927221 5800 scope.go:122] "RemoveContainer" containerID="17e8c98245387fe6cb7946bb0732d0ef7e62a0e7790e833d6017eab518bbd603" Nov 02 20:45:04.998261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:04.998232 5800 scope.go:122] "RemoveContainer" containerID="e815e1041dae2c1b23b7db6c4fd4dcbcb2c925e38a34eff4140f69dd38dc8cda" Nov 02 20:45:05.013412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.013380 5800 scope.go:122] "RemoveContainer" containerID="db796a17bcdedd08f0b4005983424afab669775eb0cafdfeb76b2eb3bb2846a1" Nov 02 20:45:05.029694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.029668 5800 scope.go:122] "RemoveContainer" containerID="67a304b7725adf70f36dbeee3f92cc755fb5b192997c768b9db2de60547c95bb" Nov 02 20:45:05.044089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.044068 5800 scope.go:122] "RemoveContainer" containerID="dbcea7fcc2447c5c107f0b29a33813d7b0bf614f84422f0181c0e65f612c9a31" Nov 02 20:45:05.059424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.059403 5800 scope.go:122] "RemoveContainer" containerID="95af79486df0b7edbe308a82e03b5c4ed81bf5e035f21a19c2ae1d3adc59696d" Nov 02 20:45:05.915423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.915378 5800 generic.go:356] "Generic (PLEG): container finished" podID="bacd99ac-5892-4452-88f6-d99c87903907" containerID="565a93594a263985158021085c080e75e62a0ec0626367e2365e543f673520d4" exitCode=60 Nov 02 20:45:05.915921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.915462 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" event={"ID":"bacd99ac-5892-4452-88f6-d99c87903907","Type":"ContainerDied","Data":"565a93594a263985158021085c080e75e62a0ec0626367e2365e543f673520d4"} Nov 02 20:45:05.915921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.915529 5800 scope.go:122] "RemoveContainer" containerID="2ae6280e65057537dbbdfe3a75588e50cb5f9b93e991663586b58e5b9a8a6a02" Nov 02 20:45:05.915921 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.915861 5800 scope.go:122] "RemoveContainer" containerID="565a93594a263985158021085c080e75e62a0ec0626367e2365e543f673520d4" Nov 02 20:45:05.916189 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.916165 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw_odh-ai-gateway-infra(bacd99ac-5892-4452-88f6-d99c87903907)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" podUID="bacd99ac-5892-4452-88f6-d99c87903907" Nov 02 20:45:05.917850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.917831 5800 scope.go:122] "RemoveContainer" containerID="735d9677c271480b8e11b4a5673b2bc22017fc3174af5b79d3c49b62e858c8a5" Nov 02 20:45:05.918214 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.918175 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8_odh-ai-gateway-infra(a697bc87-eecd-44b6-80f4-fcb0f95ae47e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" Nov 02 20:45:05.919861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.919841 5800 scope.go:122] "RemoveContainer" containerID="8d3a4fc80af7ac23a77e6bf9f949c6576c6b454a73d29692dc1ea7bffa35cc93" Nov 02 20:45:05.920268 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.920113 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r_odh-ai-gateway-infra(b04d196c-4c5a-47bb-a174-e6c399e6e1de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" Nov 02 20:45:05.922084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.922046 5800 scope.go:122] "RemoveContainer" containerID="3aaadce729d70aa38ae8137b73b2970fec6378c68c1c27ce4fbde96f22092cf9" Nov 02 20:45:05.922406 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.922381 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj_odh-ai-gateway-infra(609dc9c2-64ec-40b0-b150-6af0e275ecc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" Nov 02 20:45:05.924183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.924166 5800 scope.go:122] "RemoveContainer" containerID="6787d091af580dfda616a7ad1ec6862e438dc59547ddb150545945e34c65a9a1" Nov 02 20:45:05.924471 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.924444 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8_odh-ai-gateway-infra(b42ffa52-e457-4879-bcac-7cfcf926e29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" Nov 02 20:45:05.926312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.926264 5800 scope.go:122] "RemoveContainer" containerID="4e5babf930e0cbe82d21a719297acfffcd246fe08fa7f1a377bd806f2db7a774" Nov 02 20:45:05.926606 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.926578 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz_odh-ai-gateway-infra(25ee92b8-8bbe-4513-8973-d48c4eba73df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" Nov 02 20:45:05.928167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.928142 5800 scope.go:122] "RemoveContainer" containerID="b3ec26dd885f9682abeddd51b64e76e63b59ce2d8dd4fb633e3e1b38c6ea735c" Nov 02 20:45:05.928394 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.928362 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894205-x5zdv_odh-ai-gateway-infra(9f84c0a4-90b0-43b6-91da-b051569c7cc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" Nov 02 20:45:05.930096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.930079 5800 scope.go:122] "RemoveContainer" containerID="edf511138db652499bc10886ae7a08548c4c409b553c62d74c75fcbf95fbf489" Nov 02 20:45:05.930333 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.930308 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq_odh-ai-gateway-infra(f01c86b0-d4ba-4064-8e09-4538ffd92891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" Nov 02 20:45:05.931875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.931844 5800 scope.go:122] "RemoveContainer" containerID="bb266567ea581463bd42ab177f709539d27a4ee75929e533d2a924350c4813d3" Nov 02 20:45:05.932082 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.932058 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh_odh-ai-gateway-infra(02616515-a0a6-427a-bcc9-101c1860a2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" Nov 02 20:45:05.933509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:05.933492 5800 scope.go:122] "RemoveContainer" containerID="4aeb194aa1eb5b5493c28aecd7a4cdd8d2d9d5a823ca93bd051ad11789cd7e55" Nov 02 20:45:05.933701 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:05.933682 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd_odh-ai-gateway-infra(8ca47706-365c-42ab-b9b1-fc865aa53f6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" Nov 02 20:45:06.942466 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:06.942440 5800 scope.go:122] "RemoveContainer" containerID="565a93594a263985158021085c080e75e62a0ec0626367e2365e543f673520d4" Nov 02 20:45:06.942944 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:06.942737 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw_odh-ai-gateway-infra(bacd99ac-5892-4452-88f6-d99c87903907)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" podUID="bacd99ac-5892-4452-88f6-d99c87903907" Nov 02 20:45:15.615919 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:15.615883 5800 scope.go:122] "RemoveContainer" containerID="8d3a4fc80af7ac23a77e6bf9f949c6576c6b454a73d29692dc1ea7bffa35cc93" Nov 02 20:45:15.616459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:15.616362 5800 scope.go:122] "RemoveContainer" containerID="735d9677c271480b8e11b4a5673b2bc22017fc3174af5b79d3c49b62e858c8a5" Nov 02 20:45:16.614529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:16.614489 5800 scope.go:122] "RemoveContainer" containerID="4e5babf930e0cbe82d21a719297acfffcd246fe08fa7f1a377bd806f2db7a774" Nov 02 20:45:16.614734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:16.614567 5800 scope.go:122] "RemoveContainer" containerID="565a93594a263985158021085c080e75e62a0ec0626367e2365e543f673520d4" Nov 02 20:45:16.614734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:16.614703 5800 scope.go:122] "RemoveContainer" containerID="4aeb194aa1eb5b5493c28aecd7a4cdd8d2d9d5a823ca93bd051ad11789cd7e55" Nov 02 20:45:16.615623 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:16.614977 5800 scope.go:122] "RemoveContainer" containerID="6787d091af580dfda616a7ad1ec6862e438dc59547ddb150545945e34c65a9a1" Nov 02 20:45:17.019360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.019269 5800 generic.go:356] "Generic (PLEG): container finished" podID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerID="8852f413c3252333fab94873070a9dec975bea00bb7766932e1c8970cddea2fb" exitCode=60 Nov 02 20:45:17.019360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.019329 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" event={"ID":"a697bc87-eecd-44b6-80f4-fcb0f95ae47e","Type":"ContainerDied","Data":"8852f413c3252333fab94873070a9dec975bea00bb7766932e1c8970cddea2fb"} Nov 02 20:45:17.019864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.019432 5800 scope.go:122] "RemoveContainer" containerID="735d9677c271480b8e11b4a5673b2bc22017fc3174af5b79d3c49b62e858c8a5" Nov 02 20:45:17.019912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.019878 5800 scope.go:122] "RemoveContainer" containerID="8852f413c3252333fab94873070a9dec975bea00bb7766932e1c8970cddea2fb" Nov 02 20:45:17.020254 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:17.020229 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8_odh-ai-gateway-infra(a697bc87-eecd-44b6-80f4-fcb0f95ae47e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" Nov 02 20:45:17.022035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.022015 5800 generic.go:356] "Generic (PLEG): container finished" podID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerID="9d32f0d04fd262745dfe0a1f8f1d56f272d6e7d47c255a7445e40545db28f70f" exitCode=60 Nov 02 20:45:17.022157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.022054 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" event={"ID":"b04d196c-4c5a-47bb-a174-e6c399e6e1de","Type":"ContainerDied","Data":"9d32f0d04fd262745dfe0a1f8f1d56f272d6e7d47c255a7445e40545db28f70f"} Nov 02 20:45:17.022437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.022423 5800 scope.go:122] "RemoveContainer" containerID="9d32f0d04fd262745dfe0a1f8f1d56f272d6e7d47c255a7445e40545db28f70f" Nov 02 20:45:17.022711 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:17.022692 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r_odh-ai-gateway-infra(b04d196c-4c5a-47bb-a174-e6c399e6e1de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" Nov 02 20:45:17.034615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.034591 5800 scope.go:122] "RemoveContainer" containerID="8d3a4fc80af7ac23a77e6bf9f949c6576c6b454a73d29692dc1ea7bffa35cc93" Nov 02 20:45:17.614977 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.614945 5800 scope.go:122] "RemoveContainer" containerID="3aaadce729d70aa38ae8137b73b2970fec6378c68c1c27ce4fbde96f22092cf9" Nov 02 20:45:17.615573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.615548 5800 scope.go:122] "RemoveContainer" containerID="edf511138db652499bc10886ae7a08548c4c409b553c62d74c75fcbf95fbf489" Nov 02 20:45:17.616066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.615672 5800 scope.go:122] "RemoveContainer" containerID="bb266567ea581463bd42ab177f709539d27a4ee75929e533d2a924350c4813d3" Nov 02 20:45:17.616066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:17.615729 5800 scope.go:122] "RemoveContainer" containerID="b3ec26dd885f9682abeddd51b64e76e63b59ce2d8dd4fb633e3e1b38c6ea735c" Nov 02 20:45:18.040265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.040239 5800 generic.go:356] "Generic (PLEG): container finished" podID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerID="279b572202ff3b1a38b76aaf67a95b5a74d0dd4f647d0ba3d7c9991bdef1145e" exitCode=60 Nov 02 20:45:18.040712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.040352 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" event={"ID":"8ca47706-365c-42ab-b9b1-fc865aa53f6f","Type":"ContainerDied","Data":"279b572202ff3b1a38b76aaf67a95b5a74d0dd4f647d0ba3d7c9991bdef1145e"} Nov 02 20:45:18.040712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.040398 5800 scope.go:122] "RemoveContainer" containerID="4aeb194aa1eb5b5493c28aecd7a4cdd8d2d9d5a823ca93bd051ad11789cd7e55" Nov 02 20:45:18.040830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.040770 5800 scope.go:122] "RemoveContainer" containerID="279b572202ff3b1a38b76aaf67a95b5a74d0dd4f647d0ba3d7c9991bdef1145e" Nov 02 20:45:18.041075 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:18.041051 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd_odh-ai-gateway-infra(8ca47706-365c-42ab-b9b1-fc865aa53f6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" Nov 02 20:45:18.044447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.044427 5800 generic.go:356] "Generic (PLEG): container finished" podID="bacd99ac-5892-4452-88f6-d99c87903907" containerID="c49934bba5e20c6593f6dc11044b143e6aa37161426e3e606694fb38c3c92210" exitCode=60 Nov 02 20:45:18.044566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.044507 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" event={"ID":"bacd99ac-5892-4452-88f6-d99c87903907","Type":"ContainerDied","Data":"c49934bba5e20c6593f6dc11044b143e6aa37161426e3e606694fb38c3c92210"} Nov 02 20:45:18.044941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.044922 5800 scope.go:122] "RemoveContainer" containerID="c49934bba5e20c6593f6dc11044b143e6aa37161426e3e606694fb38c3c92210" Nov 02 20:45:18.045257 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:18.045236 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw_odh-ai-gateway-infra(bacd99ac-5892-4452-88f6-d99c87903907)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" podUID="bacd99ac-5892-4452-88f6-d99c87903907" Nov 02 20:45:18.053453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.053430 5800 generic.go:356] "Generic (PLEG): container finished" podID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerID="550061c5903a90dd8178cf9bf56698b8c0c7a8c81d48899aac32d517dff63aee" exitCode=60 Nov 02 20:45:18.053575 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.053503 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" event={"ID":"b42ffa52-e457-4879-bcac-7cfcf926e29f","Type":"ContainerDied","Data":"550061c5903a90dd8178cf9bf56698b8c0c7a8c81d48899aac32d517dff63aee"} Nov 02 20:45:18.053875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.053859 5800 scope.go:122] "RemoveContainer" containerID="550061c5903a90dd8178cf9bf56698b8c0c7a8c81d48899aac32d517dff63aee" Nov 02 20:45:18.054162 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:18.054141 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8_odh-ai-gateway-infra(b42ffa52-e457-4879-bcac-7cfcf926e29f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" Nov 02 20:45:18.056422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.056398 5800 generic.go:356] "Generic (PLEG): container finished" podID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerID="102aa91a7d9ecf0836be2b20e0d7686324f33ebd80df2da6463bb79228fc8a48" exitCode=60 Nov 02 20:45:18.056510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.056466 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" event={"ID":"25ee92b8-8bbe-4513-8973-d48c4eba73df","Type":"ContainerDied","Data":"102aa91a7d9ecf0836be2b20e0d7686324f33ebd80df2da6463bb79228fc8a48"} Nov 02 20:45:18.056861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.056846 5800 scope.go:122] "RemoveContainer" containerID="102aa91a7d9ecf0836be2b20e0d7686324f33ebd80df2da6463bb79228fc8a48" Nov 02 20:45:18.057148 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:18.057129 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz_odh-ai-gateway-infra(25ee92b8-8bbe-4513-8973-d48c4eba73df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" Nov 02 20:45:18.057997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.057979 5800 scope.go:122] "RemoveContainer" containerID="565a93594a263985158021085c080e75e62a0ec0626367e2365e543f673520d4" Nov 02 20:45:18.075840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.075815 5800 scope.go:122] "RemoveContainer" containerID="6787d091af580dfda616a7ad1ec6862e438dc59547ddb150545945e34c65a9a1" Nov 02 20:45:18.092555 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.092528 5800 scope.go:122] "RemoveContainer" containerID="4e5babf930e0cbe82d21a719297acfffcd246fe08fa7f1a377bd806f2db7a774" Nov 02 20:45:18.136733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.136700 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8"] Nov 02 20:45:18.250964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.250922 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r"] Nov 02 20:45:18.363132 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.363101 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:18.434291 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.434251 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:18.463186 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.463144 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-kube-api-access-2dvvq\" (UniqueName: \"kubernetes.io/projected/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-kube-api-access-2dvvq\") pod \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " Nov 02 20:45:18.463409 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.463254 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-maas-api-service-ca\") pod \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\" (UID: \"a697bc87-eecd-44b6-80f4-fcb0f95ae47e\") " Nov 02 20:45:18.463676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.463647 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-maas-api-service-ca" pod "a697bc87-eecd-44b6-80f4-fcb0f95ae47e" (UID: "a697bc87-eecd-44b6-80f4-fcb0f95ae47e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:18.463929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.463910 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:18.526065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.526003 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-kube-api-access-2dvvq" pod "a697bc87-eecd-44b6-80f4-fcb0f95ae47e" (UID: "a697bc87-eecd-44b6-80f4-fcb0f95ae47e"). InnerVolumeSpecName "kube-api-access-2dvvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:18.565143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.565110 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04d196c-4c5a-47bb-a174-e6c399e6e1de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04d196c-4c5a-47bb-a174-e6c399e6e1de-maas-api-service-ca\") pod \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " Nov 02 20:45:18.565373 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.565207 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04d196c-4c5a-47bb-a174-e6c399e6e1de-kube-api-access-t9qbm\" (UniqueName: \"kubernetes.io/projected/b04d196c-4c5a-47bb-a174-e6c399e6e1de-kube-api-access-t9qbm\") pod \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\" (UID: \"b04d196c-4c5a-47bb-a174-e6c399e6e1de\") " Nov 02 20:45:18.565498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.565474 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04d196c-4c5a-47bb-a174-e6c399e6e1de-maas-api-service-ca" pod "b04d196c-4c5a-47bb-a174-e6c399e6e1de" (UID: "b04d196c-4c5a-47bb-a174-e6c399e6e1de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:18.565806 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.565784 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvvq\" (UniqueName: \"kubernetes.io/projected/a697bc87-eecd-44b6-80f4-fcb0f95ae47e-kube-api-access-2dvvq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:18.565883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.565809 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b04d196c-4c5a-47bb-a174-e6c399e6e1de-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:18.567510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.567487 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04d196c-4c5a-47bb-a174-e6c399e6e1de-kube-api-access-t9qbm" pod "b04d196c-4c5a-47bb-a174-e6c399e6e1de" (UID: "b04d196c-4c5a-47bb-a174-e6c399e6e1de"). InnerVolumeSpecName "kube-api-access-t9qbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:18.667522 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:18.667477 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9qbm\" (UniqueName: \"kubernetes.io/projected/b04d196c-4c5a-47bb-a174-e6c399e6e1de-kube-api-access-t9qbm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.073406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.073372 5800 generic.go:356] "Generic (PLEG): container finished" podID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerID="81f1f6de2379a7329d4e337393fce8f5786f4f50e16e459bb61f8e781bc9c5fa" exitCode=60 Nov 02 20:45:19.073928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.073446 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" event={"ID":"f01c86b0-d4ba-4064-8e09-4538ffd92891","Type":"ContainerDied","Data":"81f1f6de2379a7329d4e337393fce8f5786f4f50e16e459bb61f8e781bc9c5fa"} Nov 02 20:45:19.073928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.073552 5800 scope.go:122] "RemoveContainer" containerID="edf511138db652499bc10886ae7a08548c4c409b553c62d74c75fcbf95fbf489" Nov 02 20:45:19.074066 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.073947 5800 scope.go:122] "RemoveContainer" containerID="81f1f6de2379a7329d4e337393fce8f5786f4f50e16e459bb61f8e781bc9c5fa" Nov 02 20:45:19.074604 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:19.074233 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq_odh-ai-gateway-infra(f01c86b0-d4ba-4064-8e09-4538ffd92891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" Nov 02 20:45:19.075678 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.075652 5800 generic.go:356] "Generic (PLEG): container finished" podID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerID="2aa5bc2b5600c40b4144a98e165ddbc959dfc3bd75804487cf4b309ef85ca8b3" exitCode=60 Nov 02 20:45:19.075809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.075721 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" event={"ID":"02616515-a0a6-427a-bcc9-101c1860a2d7","Type":"ContainerDied","Data":"2aa5bc2b5600c40b4144a98e165ddbc959dfc3bd75804487cf4b309ef85ca8b3"} Nov 02 20:45:19.076077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.076054 5800 scope.go:122] "RemoveContainer" containerID="2aa5bc2b5600c40b4144a98e165ddbc959dfc3bd75804487cf4b309ef85ca8b3" Nov 02 20:45:19.076315 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:19.076275 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh_odh-ai-gateway-infra(02616515-a0a6-427a-bcc9-101c1860a2d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" Nov 02 20:45:19.080328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.080276 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" Nov 02 20:45:19.080413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.080320 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8" event={"ID":"a697bc87-eecd-44b6-80f4-fcb0f95ae47e","Type":"ContainerDied","Data":"a66d1551a3dd9f0e04eb6852dc9fe85e32bf5aa9f8d95b51c4b3ded66abdd4f3"} Nov 02 20:45:19.081936 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.081916 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" event={"ID":"b04d196c-4c5a-47bb-a174-e6c399e6e1de","Type":"ContainerDied","Data":"f0bed840f4d5061d32601cfe47bb67ef48d03e88b2e4c97481a5b56f5ecf881d"} Nov 02 20:45:19.082343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.082271 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r" Nov 02 20:45:19.083962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.083941 5800 generic.go:356] "Generic (PLEG): container finished" podID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerID="79654385ee8ef7e03df91e863b1e08555d7f6bf38fb70c3dd0048c7d52cbe5f3" exitCode=60 Nov 02 20:45:19.084244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.084007 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" event={"ID":"609dc9c2-64ec-40b0-b150-6af0e275ecc5","Type":"ContainerDied","Data":"79654385ee8ef7e03df91e863b1e08555d7f6bf38fb70c3dd0048c7d52cbe5f3"} Nov 02 20:45:19.084598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.084583 5800 scope.go:122] "RemoveContainer" containerID="79654385ee8ef7e03df91e863b1e08555d7f6bf38fb70c3dd0048c7d52cbe5f3" Nov 02 20:45:19.084795 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:19.084772 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj_odh-ai-gateway-infra(609dc9c2-64ec-40b0-b150-6af0e275ecc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" Nov 02 20:45:19.090437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.090417 5800 scope.go:122] "RemoveContainer" containerID="bb266567ea581463bd42ab177f709539d27a4ee75929e533d2a924350c4813d3" Nov 02 20:45:19.093007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.092979 5800 generic.go:356] "Generic (PLEG): container finished" podID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerID="cbd1b81ddcf4bde7c663795cb1e23a41fed731e16330aee410549c85a8817dba" exitCode=60 Nov 02 20:45:19.093111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.093016 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" event={"ID":"9f84c0a4-90b0-43b6-91da-b051569c7cc2","Type":"ContainerDied","Data":"cbd1b81ddcf4bde7c663795cb1e23a41fed731e16330aee410549c85a8817dba"} Nov 02 20:45:19.093718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.093517 5800 scope.go:122] "RemoveContainer" containerID="cbd1b81ddcf4bde7c663795cb1e23a41fed731e16330aee410549c85a8817dba" Nov 02 20:45:19.094242 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:19.093812 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894205-x5zdv_odh-ai-gateway-infra(9f84c0a4-90b0-43b6-91da-b051569c7cc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" Nov 02 20:45:19.108163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.108143 5800 scope.go:122] "RemoveContainer" containerID="8852f413c3252333fab94873070a9dec975bea00bb7766932e1c8970cddea2fb" Nov 02 20:45:19.118111 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.118091 5800 scope.go:122] "RemoveContainer" containerID="9d32f0d04fd262745dfe0a1f8f1d56f272d6e7d47c255a7445e40545db28f70f" Nov 02 20:45:19.127787 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.127768 5800 scope.go:122] "RemoveContainer" containerID="3aaadce729d70aa38ae8137b73b2970fec6378c68c1c27ce4fbde96f22092cf9" Nov 02 20:45:19.141879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.141848 5800 scope.go:122] "RemoveContainer" containerID="b3ec26dd885f9682abeddd51b64e76e63b59ce2d8dd4fb633e3e1b38c6ea735c" Nov 02 20:45:19.160258 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.160232 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd"] Nov 02 20:45:19.269359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.269323 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz"] Nov 02 20:45:19.306349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.306323 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:19.374989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.374818 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw"] Nov 02 20:45:19.382851 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.382823 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca47706-365c-42ab-b9b1-fc865aa53f6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca47706-365c-42ab-b9b1-fc865aa53f6f-maas-api-service-ca\") pod \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " Nov 02 20:45:19.382995 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.382880 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca47706-365c-42ab-b9b1-fc865aa53f6f-kube-api-access-fxnnz\" (UniqueName: \"kubernetes.io/projected/8ca47706-365c-42ab-b9b1-fc865aa53f6f-kube-api-access-fxnnz\") pod \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\" (UID: \"8ca47706-365c-42ab-b9b1-fc865aa53f6f\") " Nov 02 20:45:19.383246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.383216 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca47706-365c-42ab-b9b1-fc865aa53f6f-maas-api-service-ca" pod "8ca47706-365c-42ab-b9b1-fc865aa53f6f" (UID: "8ca47706-365c-42ab-b9b1-fc865aa53f6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:19.383474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.383453 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca47706-365c-42ab-b9b1-fc865aa53f6f-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.385637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.385607 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca47706-365c-42ab-b9b1-fc865aa53f6f-kube-api-access-fxnnz" pod "8ca47706-365c-42ab-b9b1-fc865aa53f6f" (UID: "8ca47706-365c-42ab-b9b1-fc865aa53f6f"). InnerVolumeSpecName "kube-api-access-fxnnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:19.422560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.422533 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:19.484908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.484870 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxnnz\" (UniqueName: \"kubernetes.io/projected/8ca47706-365c-42ab-b9b1-fc865aa53f6f-kube-api-access-fxnnz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.536818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.535642 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:19.587362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.585624 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ee92b8-8bbe-4513-8973-d48c4eba73df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ee92b8-8bbe-4513-8973-d48c4eba73df-maas-api-service-ca\") pod \"25ee92b8-8bbe-4513-8973-d48c4eba73df\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " Nov 02 20:45:19.587362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.585833 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ee92b8-8bbe-4513-8973-d48c4eba73df-kube-api-access-5j756\" (UniqueName: \"kubernetes.io/projected/25ee92b8-8bbe-4513-8973-d48c4eba73df-kube-api-access-5j756\") pod \"25ee92b8-8bbe-4513-8973-d48c4eba73df\" (UID: \"25ee92b8-8bbe-4513-8973-d48c4eba73df\") " Nov 02 20:45:19.589975 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.588763 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ee92b8-8bbe-4513-8973-d48c4eba73df-maas-api-service-ca" pod "25ee92b8-8bbe-4513-8973-d48c4eba73df" (UID: "25ee92b8-8bbe-4513-8973-d48c4eba73df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:19.593604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.593573 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ee92b8-8bbe-4513-8973-d48c4eba73df-kube-api-access-5j756" pod "25ee92b8-8bbe-4513-8973-d48c4eba73df" (UID: "25ee92b8-8bbe-4513-8973-d48c4eba73df"). InnerVolumeSpecName "kube-api-access-5j756". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:19.596272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.596246 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8"] Nov 02 20:45:19.608673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.608643 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8"] Nov 02 20:45:19.613677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.613647 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894205-ppsx8"] Nov 02 20:45:19.630325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.627070 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" path="/var/lib/kubelet/pods/a697bc87-eecd-44b6-80f4-fcb0f95ae47e/volumes" Nov 02 20:45:19.687640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.687606 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacd99ac-5892-4452-88f6-d99c87903907-kube-api-access-74j5l\" (UniqueName: \"kubernetes.io/projected/bacd99ac-5892-4452-88f6-d99c87903907-kube-api-access-74j5l\") pod \"bacd99ac-5892-4452-88f6-d99c87903907\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " Nov 02 20:45:19.687795 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.687743 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacd99ac-5892-4452-88f6-d99c87903907-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacd99ac-5892-4452-88f6-d99c87903907-maas-api-service-ca\") pod \"bacd99ac-5892-4452-88f6-d99c87903907\" (UID: \"bacd99ac-5892-4452-88f6-d99c87903907\") " Nov 02 20:45:19.688364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.688333 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacd99ac-5892-4452-88f6-d99c87903907-maas-api-service-ca" pod "bacd99ac-5892-4452-88f6-d99c87903907" (UID: "bacd99ac-5892-4452-88f6-d99c87903907"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:19.688904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.688881 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ee92b8-8bbe-4513-8973-d48c4eba73df-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.689051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.689033 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j756\" (UniqueName: \"kubernetes.io/projected/25ee92b8-8bbe-4513-8973-d48c4eba73df-kube-api-access-5j756\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.689117 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.689055 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bacd99ac-5892-4452-88f6-d99c87903907-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.692321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.691971 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacd99ac-5892-4452-88f6-d99c87903907-kube-api-access-74j5l" pod "bacd99ac-5892-4452-88f6-d99c87903907" (UID: "bacd99ac-5892-4452-88f6-d99c87903907"). InnerVolumeSpecName "kube-api-access-74j5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:19.719689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.719615 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r"] Nov 02 20:45:19.723203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.723173 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6a4ab0-29894205-pgt2r"] Nov 02 20:45:19.790347 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.790320 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74j5l\" (UniqueName: \"kubernetes.io/projected/bacd99ac-5892-4452-88f6-d99c87903907-kube-api-access-74j5l\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.790541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.790522 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:19.891121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.891031 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42ffa52-e457-4879-bcac-7cfcf926e29f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ffa52-e457-4879-bcac-7cfcf926e29f-maas-api-service-ca\") pod \"b42ffa52-e457-4879-bcac-7cfcf926e29f\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " Nov 02 20:45:19.891121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.891107 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42ffa52-e457-4879-bcac-7cfcf926e29f-kube-api-access-n5drm\" (UniqueName: \"kubernetes.io/projected/b42ffa52-e457-4879-bcac-7cfcf926e29f-kube-api-access-n5drm\") pod \"b42ffa52-e457-4879-bcac-7cfcf926e29f\" (UID: \"b42ffa52-e457-4879-bcac-7cfcf926e29f\") " Nov 02 20:45:19.891477 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.891453 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ffa52-e457-4879-bcac-7cfcf926e29f-maas-api-service-ca" pod "b42ffa52-e457-4879-bcac-7cfcf926e29f" (UID: "b42ffa52-e457-4879-bcac-7cfcf926e29f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:19.891749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.891729 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ffa52-e457-4879-bcac-7cfcf926e29f-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:19.893325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.893298 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42ffa52-e457-4879-bcac-7cfcf926e29f-kube-api-access-n5drm" pod "b42ffa52-e457-4879-bcac-7cfcf926e29f" (UID: "b42ffa52-e457-4879-bcac-7cfcf926e29f"). InnerVolumeSpecName "kube-api-access-n5drm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:19.992983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:19.992939 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5drm\" (UniqueName: \"kubernetes.io/projected/b42ffa52-e457-4879-bcac-7cfcf926e29f-kube-api-access-n5drm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.111374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.111347 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" Nov 02 20:45:20.111374 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.111358 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd" event={"ID":"8ca47706-365c-42ab-b9b1-fc865aa53f6f","Type":"ContainerDied","Data":"0e80d38ee42ca6b04495757d0adb2cd04938ad02855a8db052c99e025a903f0d"} Nov 02 20:45:20.111899 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.111418 5800 scope.go:122] "RemoveContainer" containerID="279b572202ff3b1a38b76aaf67a95b5a74d0dd4f647d0ba3d7c9991bdef1145e" Nov 02 20:45:20.112948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.112930 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" Nov 02 20:45:20.113065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.112942 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw" event={"ID":"bacd99ac-5892-4452-88f6-d99c87903907","Type":"ContainerDied","Data":"d782db716d2d4c9a446493ad1ef1a529e3af68ce2a466d347bd0df5182c03883"} Nov 02 20:45:20.118563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.118534 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" Nov 02 20:45:20.118563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.118538 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8" event={"ID":"b42ffa52-e457-4879-bcac-7cfcf926e29f","Type":"ContainerDied","Data":"c6764f767c7234e0d3d0b456e8f5d22c4835ccfcbb2c5fc6f5c1cbf4a82a953a"} Nov 02 20:45:20.119908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.119891 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" Nov 02 20:45:20.120019 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.119942 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz" event={"ID":"25ee92b8-8bbe-4513-8973-d48c4eba73df","Type":"ContainerDied","Data":"dc7db7b7fea1b6cb6f74d1fbe2f6402a56421d990735be82eca57dcdf0f9b898"} Nov 02 20:45:20.124577 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.124556 5800 scope.go:122] "RemoveContainer" containerID="c49934bba5e20c6593f6dc11044b143e6aa37161426e3e606694fb38c3c92210" Nov 02 20:45:20.135038 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.135015 5800 scope.go:122] "RemoveContainer" containerID="550061c5903a90dd8178cf9bf56698b8c0c7a8c81d48899aac32d517dff63aee" Nov 02 20:45:20.144236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.144212 5800 scope.go:122] "RemoveContainer" containerID="102aa91a7d9ecf0836be2b20e0d7686324f33ebd80df2da6463bb79228fc8a48" Nov 02 20:45:20.193848 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.193811 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq"] Nov 02 20:45:20.330849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.330823 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:20.376067 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.376032 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd"] Nov 02 20:45:20.385954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.385925 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c25349-29894205-jrffd"] Nov 02 20:45:20.406591 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.406516 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj"] Nov 02 20:45:20.492166 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.492133 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8"] Nov 02 20:45:20.496523 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.496492 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-78bef7-29894205-sdhw8"] Nov 02 20:45:20.501180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.501151 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01c86b0-d4ba-4064-8e09-4538ffd92891-kube-api-access-c9xfw\" (UniqueName: \"kubernetes.io/projected/f01c86b0-d4ba-4064-8e09-4538ffd92891-kube-api-access-c9xfw\") pod \"f01c86b0-d4ba-4064-8e09-4538ffd92891\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " Nov 02 20:45:20.501348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.501245 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01c86b0-d4ba-4064-8e09-4538ffd92891-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01c86b0-d4ba-4064-8e09-4538ffd92891-maas-api-service-ca\") pod \"f01c86b0-d4ba-4064-8e09-4538ffd92891\" (UID: \"f01c86b0-d4ba-4064-8e09-4538ffd92891\") " Nov 02 20:45:20.501622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.501596 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01c86b0-d4ba-4064-8e09-4538ffd92891-maas-api-service-ca" pod "f01c86b0-d4ba-4064-8e09-4538ffd92891" (UID: "f01c86b0-d4ba-4064-8e09-4538ffd92891"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:20.504537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.504513 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01c86b0-d4ba-4064-8e09-4538ffd92891-kube-api-access-c9xfw" pod "f01c86b0-d4ba-4064-8e09-4538ffd92891" (UID: "f01c86b0-d4ba-4064-8e09-4538ffd92891"). InnerVolumeSpecName "kube-api-access-c9xfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:20.512362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.512334 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh"] Nov 02 20:45:20.547557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.547534 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:20.598343 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.597559 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz"] Nov 02 20:45:20.602222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.602179 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894205-qgdbz"] Nov 02 20:45:20.603531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.603502 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xfw\" (UniqueName: \"kubernetes.io/projected/f01c86b0-d4ba-4064-8e09-4538ffd92891-kube-api-access-c9xfw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.603636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.603536 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f01c86b0-d4ba-4064-8e09-4538ffd92891-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.695352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.695260 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:20.704044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.704020 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609dc9c2-64ec-40b0-b150-6af0e275ecc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/609dc9c2-64ec-40b0-b150-6af0e275ecc5-maas-api-service-ca\") pod \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " Nov 02 20:45:20.704231 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.704207 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609dc9c2-64ec-40b0-b150-6af0e275ecc5-kube-api-access-jrgrv\" (UniqueName: \"kubernetes.io/projected/609dc9c2-64ec-40b0-b150-6af0e275ecc5-kube-api-access-jrgrv\") pod \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\" (UID: \"609dc9c2-64ec-40b0-b150-6af0e275ecc5\") " Nov 02 20:45:20.704413 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.704394 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609dc9c2-64ec-40b0-b150-6af0e275ecc5-maas-api-service-ca" pod "609dc9c2-64ec-40b0-b150-6af0e275ecc5" (UID: "609dc9c2-64ec-40b0-b150-6af0e275ecc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:20.704788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.704757 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/609dc9c2-64ec-40b0-b150-6af0e275ecc5-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.706777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.706748 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609dc9c2-64ec-40b0-b150-6af0e275ecc5-kube-api-access-jrgrv" pod "609dc9c2-64ec-40b0-b150-6af0e275ecc5" (UID: "609dc9c2-64ec-40b0-b150-6af0e275ecc5"). InnerVolumeSpecName "kube-api-access-jrgrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:20.709275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.709026 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw"] Nov 02 20:45:20.713857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.713836 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894205-z6jsw"] Nov 02 20:45:20.805521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.805488 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02616515-a0a6-427a-bcc9-101c1860a2d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02616515-a0a6-427a-bcc9-101c1860a2d7-maas-api-service-ca\") pod \"02616515-a0a6-427a-bcc9-101c1860a2d7\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " Nov 02 20:45:20.805708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.805579 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02616515-a0a6-427a-bcc9-101c1860a2d7-kube-api-access-65hww\" (UniqueName: \"kubernetes.io/projected/02616515-a0a6-427a-bcc9-101c1860a2d7-kube-api-access-65hww\") pod \"02616515-a0a6-427a-bcc9-101c1860a2d7\" (UID: \"02616515-a0a6-427a-bcc9-101c1860a2d7\") " Nov 02 20:45:20.805882 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.805869 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrgrv\" (UniqueName: \"kubernetes.io/projected/609dc9c2-64ec-40b0-b150-6af0e275ecc5-kube-api-access-jrgrv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.805944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.805917 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02616515-a0a6-427a-bcc9-101c1860a2d7-maas-api-service-ca" pod "02616515-a0a6-427a-bcc9-101c1860a2d7" (UID: "02616515-a0a6-427a-bcc9-101c1860a2d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:20.807715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.807681 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02616515-a0a6-427a-bcc9-101c1860a2d7-kube-api-access-65hww" pod "02616515-a0a6-427a-bcc9-101c1860a2d7" (UID: "02616515-a0a6-427a-bcc9-101c1860a2d7"). InnerVolumeSpecName "kube-api-access-65hww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:20.851948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.851911 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv"] Nov 02 20:45:20.906961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.906929 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hww\" (UniqueName: \"kubernetes.io/projected/02616515-a0a6-427a-bcc9-101c1860a2d7-kube-api-access-65hww\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:20.906961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:20.906959 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02616515-a0a6-427a-bcc9-101c1860a2d7-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.019081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.019056 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:21.109190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.109153 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f84c0a4-90b0-43b6-91da-b051569c7cc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f84c0a4-90b0-43b6-91da-b051569c7cc2-maas-api-service-ca\") pod \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " Nov 02 20:45:21.109412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.109318 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f84c0a4-90b0-43b6-91da-b051569c7cc2-kube-api-access-bmb6m\" (UniqueName: \"kubernetes.io/projected/9f84c0a4-90b0-43b6-91da-b051569c7cc2-kube-api-access-bmb6m\") pod \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\" (UID: \"9f84c0a4-90b0-43b6-91da-b051569c7cc2\") " Nov 02 20:45:21.109574 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.109549 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f84c0a4-90b0-43b6-91da-b051569c7cc2-maas-api-service-ca" pod "9f84c0a4-90b0-43b6-91da-b051569c7cc2" (UID: "9f84c0a4-90b0-43b6-91da-b051569c7cc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:45:21.109769 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.109755 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f84c0a4-90b0-43b6-91da-b051569c7cc2-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.111329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.111307 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f84c0a4-90b0-43b6-91da-b051569c7cc2-kube-api-access-bmb6m" pod "9f84c0a4-90b0-43b6-91da-b051569c7cc2" (UID: "9f84c0a4-90b0-43b6-91da-b051569c7cc2"). InnerVolumeSpecName "kube-api-access-bmb6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:21.130393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.130365 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" Nov 02 20:45:21.130393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.130377 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv" event={"ID":"9f84c0a4-90b0-43b6-91da-b051569c7cc2","Type":"ContainerDied","Data":"dcc8791982445ebc90e544d9431eb39671c45b381e4939c63da888493085a9e6"} Nov 02 20:45:21.130912 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.130444 5800 scope.go:122] "RemoveContainer" containerID="cbd1b81ddcf4bde7c663795cb1e23a41fed731e16330aee410549c85a8817dba" Nov 02 20:45:21.136698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.136361 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" Nov 02 20:45:21.136698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.136373 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq" event={"ID":"f01c86b0-d4ba-4064-8e09-4538ffd92891","Type":"ContainerDied","Data":"5bc95ee982ee10693c95870109d9d739f9a00c2803ecfd737f12629c4bc3cceb"} Nov 02 20:45:21.138202 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.138173 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" event={"ID":"02616515-a0a6-427a-bcc9-101c1860a2d7","Type":"ContainerDied","Data":"6f17176d4e473af59251b2d78de35814510f5eef7576668b25cf49ea93cccefb"} Nov 02 20:45:21.138376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.138331 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh" Nov 02 20:45:21.154931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.154908 5800 scope.go:122] "RemoveContainer" containerID="81f1f6de2379a7329d4e337393fce8f5786f4f50e16e459bb61f8e781bc9c5fa" Nov 02 20:45:21.156886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.156861 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" Nov 02 20:45:21.156989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.156867 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj" event={"ID":"609dc9c2-64ec-40b0-b150-6af0e275ecc5","Type":"ContainerDied","Data":"9d3c49a3c014bd7903721305f6ad874daa68dcdda9601893aff1245b22e282f7"} Nov 02 20:45:21.170818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.170792 5800 scope.go:122] "RemoveContainer" containerID="2aa5bc2b5600c40b4144a98e165ddbc959dfc3bd75804487cf4b309ef85ca8b3" Nov 02 20:45:21.184868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.184827 5800 scope.go:122] "RemoveContainer" containerID="79654385ee8ef7e03df91e863b1e08555d7f6bf38fb70c3dd0048c7d52cbe5f3" Nov 02 20:45:21.210681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.210647 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmb6m\" (UniqueName: \"kubernetes.io/projected/9f84c0a4-90b0-43b6-91da-b051569c7cc2-kube-api-access-bmb6m\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:21.350659 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.350616 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj"] Nov 02 20:45:21.354473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.354445 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894205-k7xrj"] Nov 02 20:45:21.458387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.458350 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh"] Nov 02 20:45:21.463165 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.463140 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894205-6mtqh"] Nov 02 20:45:21.560104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.560071 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq"] Nov 02 20:45:21.567762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.567730 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894205-vt6nq"] Nov 02 20:45:21.623503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.623416 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" path="/var/lib/kubelet/pods/02616515-a0a6-427a-bcc9-101c1860a2d7/volumes" Nov 02 20:45:21.624042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.624018 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" path="/var/lib/kubelet/pods/25ee92b8-8bbe-4513-8973-d48c4eba73df/volumes" Nov 02 20:45:21.624555 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.624535 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" path="/var/lib/kubelet/pods/609dc9c2-64ec-40b0-b150-6af0e275ecc5/volumes" Nov 02 20:45:21.625500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.625478 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" path="/var/lib/kubelet/pods/8ca47706-365c-42ab-b9b1-fc865aa53f6f/volumes" Nov 02 20:45:21.625944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.625923 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" path="/var/lib/kubelet/pods/b04d196c-4c5a-47bb-a174-e6c399e6e1de/volumes" Nov 02 20:45:21.626408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.626390 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" path="/var/lib/kubelet/pods/b42ffa52-e457-4879-bcac-7cfcf926e29f/volumes" Nov 02 20:45:21.627033 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.627014 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacd99ac-5892-4452-88f6-d99c87903907" path="/var/lib/kubelet/pods/bacd99ac-5892-4452-88f6-d99c87903907/volumes" Nov 02 20:45:21.627476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.627460 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" path="/var/lib/kubelet/pods/f01c86b0-d4ba-4064-8e09-4538ffd92891/volumes" Nov 02 20:45:21.668726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.668687 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv"] Nov 02 20:45:21.674423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:21.674396 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894205-x5zdv"] Nov 02 20:45:22.697513 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:22.697475 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:45:23.626377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:23.626341 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" path="/var/lib/kubelet/pods/9f84c0a4-90b0-43b6-91da-b051569c7cc2/volumes" Nov 02 20:45:24.340625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:24.340588 5800 scope.go:122] "RemoveContainer" containerID="54e9bfa1bdade7189180a110442e43edb7d11b2b8c3acc9b6c9bb59b5e28332f" Nov 02 20:45:40.070789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.070751 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl"] Nov 02 20:45:40.071357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071335 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.071357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071348 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.071357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071358 5800 cpu_manager.go:414] "containerMap: removing container" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071364 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071374 5800 cpu_manager.go:414] "containerMap: removing container" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071380 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071386 5800 cpu_manager.go:414] "containerMap: removing container" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071391 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071400 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071404 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071410 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071415 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071427 5800 cpu_manager.go:414] "containerMap: removing container" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071431 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071436 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071442 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071448 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071453 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071463 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071468 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071476 5800 cpu_manager.go:414] "containerMap: removing container" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071481 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071488 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071494 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071499 5800 cpu_manager.go:414] "containerMap: removing container" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071503 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.071497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071509 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071514 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071521 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071525 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071538 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071543 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071550 5800 cpu_manager.go:414] "containerMap: removing container" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071554 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071562 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071566 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071574 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071579 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071585 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071590 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071602 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071607 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071614 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071618 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071624 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071628 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071638 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071642 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071648 5800 cpu_manager.go:414] "containerMap: removing container" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071652 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071658 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071662 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071743 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071750 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071757 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071763 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071770 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071777 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071783 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071787 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071791 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071796 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42ffa52-e457-4879-bcac-7cfcf926e29f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071800 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071805 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071809 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071815 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071820 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071825 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071830 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071836 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071842 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071847 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca47706-365c-42ab-b9b1-fc865aa53f6f" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071852 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071859 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071864 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a697bc87-eecd-44b6-80f4-fcb0f95ae47e" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071871 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="609dc9c2-64ec-40b0-b150-6af0e275ecc5" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071876 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.072352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.071881 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.076975 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.076954 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.107586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.107556 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393642a7-a1cb-4e36-abda-e77651962708-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.107727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.107691 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpg9\" (UniqueName: \"kubernetes.io/projected/393642a7-a1cb-4e36-abda-e77651962708-kube-api-access-mkpg9\") pod \"payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.139941 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.136810 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl"] Nov 02 20:45:40.170666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.170633 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0a8f62\"" Nov 02 20:45:40.171557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.171529 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0a8f62-dockercfg-76wfc\"" Nov 02 20:45:40.212855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.208596 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpg9\" (UniqueName: \"kubernetes.io/projected/393642a7-a1cb-4e36-abda-e77651962708-kube-api-access-mkpg9\") pod \"payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.212855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.208726 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393642a7-a1cb-4e36-abda-e77651962708-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.212855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.209496 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393642a7-a1cb-4e36-abda-e77651962708-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.235947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.235917 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9"] Nov 02 20:45:40.236568 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236552 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.236626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236570 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.236626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236587 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.236626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236592 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f84c0a4-90b0-43b6-91da-b051569c7cc2" containerName="cleanup" Nov 02 20:45:40.236626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236614 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.236626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236619 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.236816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236631 5800 cpu_manager.go:414] "containerMap: removing container" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.236816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236636 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ee92b8-8bbe-4513-8973-d48c4eba73df" containerName="cleanup" Nov 02 20:45:40.236816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236741 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="02616515-a0a6-427a-bcc9-101c1860a2d7" containerName="cleanup" Nov 02 20:45:40.236816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236748 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01c86b0-d4ba-4064-8e09-4538ffd92891" containerName="cleanup" Nov 02 20:45:40.236816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236756 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacd99ac-5892-4452-88f6-d99c87903907" containerName="cleanup" Nov 02 20:45:40.236816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.236762 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04d196c-4c5a-47bb-a174-e6c399e6e1de" containerName="cleanup" Nov 02 20:45:40.240486 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.240469 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.263162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.263134 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpg9\" (UniqueName: \"kubernetes.io/projected/393642a7-a1cb-4e36-abda-e77651962708-kube-api-access-mkpg9\") pod \"payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.294216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.294184 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9"] Nov 02 20:45:40.310422 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.310387 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rbr\" (UniqueName: \"kubernetes.io/projected/6a016fb4-e940-4d5c-a282-b541fc3957c2-kube-api-access-22rbr\") pod \"payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.310604 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.310498 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a016fb4-e940-4d5c-a282-b541fc3957c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.391973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.391891 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:40.411752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.411720 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rbr\" (UniqueName: \"kubernetes.io/projected/6a016fb4-e940-4d5c-a282-b541fc3957c2-kube-api-access-22rbr\") pod \"payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.411903 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.411802 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a016fb4-e940-4d5c-a282-b541fc3957c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.412861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.412834 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a016fb4-e940-4d5c-a282-b541fc3957c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.474718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.474682 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rbr\" (UniqueName: \"kubernetes.io/projected/6a016fb4-e940-4d5c-a282-b541fc3957c2-kube-api-access-22rbr\") pod \"payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.555699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.555658 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:40.767446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.767418 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl"] Nov 02 20:45:40.768783 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:40.768755 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393642a7_a1cb_4e36_abda_e77651962708.slice/crio-748bfa2a831a9c722568886cf9c7b483c1a302fd0969c9533d58a7439fcfbd5f WatchSource:0}: Error finding container 748bfa2a831a9c722568886cf9c7b483c1a302fd0969c9533d58a7439fcfbd5f: Status 404 returned error can't find the container with id 748bfa2a831a9c722568886cf9c7b483c1a302fd0969c9533d58a7439fcfbd5f Nov 02 20:45:40.936557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:40.936531 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9"] Nov 02 20:45:40.937305 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:40.937256 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a016fb4_e940_4d5c_a282_b541fc3957c2.slice/crio-f231df43a8c003fa53e32a6dcb7763589a3d09bdfa83c58d504f17c20699afd9 WatchSource:0}: Error finding container f231df43a8c003fa53e32a6dcb7763589a3d09bdfa83c58d504f17c20699afd9: Status 404 returned error can't find the container with id f231df43a8c003fa53e32a6dcb7763589a3d09bdfa83c58d504f17c20699afd9 Nov 02 20:45:41.311338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.311252 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" event={"ID":"6a016fb4-e940-4d5c-a282-b541fc3957c2","Type":"ContainerStarted","Data":"5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf"} Nov 02 20:45:41.311338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.311338 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" event={"ID":"6a016fb4-e940-4d5c-a282-b541fc3957c2","Type":"ContainerStarted","Data":"f231df43a8c003fa53e32a6dcb7763589a3d09bdfa83c58d504f17c20699afd9"} Nov 02 20:45:41.311830 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.311367 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:41.312894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.312872 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" event={"ID":"393642a7-a1cb-4e36-abda-e77651962708","Type":"ContainerStarted","Data":"c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3"} Nov 02 20:45:41.312967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.312913 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" event={"ID":"393642a7-a1cb-4e36-abda-e77651962708","Type":"ContainerStarted","Data":"748bfa2a831a9c722568886cf9c7b483c1a302fd0969c9533d58a7439fcfbd5f"} Nov 02 20:45:41.312967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.312955 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:41.421491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.421431 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" podStartSLOduration=1.421416089 podStartE2EDuration="1.421416089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:41.41905307 +0000 UTC m=+3439.935825654" watchObservedRunningTime="2026-11-02 20:45:41.421416089 +0000 UTC m=+3439.938188668" Nov 02 20:45:41.542917 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:41.526345 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" podStartSLOduration=1.526326821 podStartE2EDuration="1.526326821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:41.525993054 +0000 UTC m=+3440.042765651" watchObservedRunningTime="2026-11-02 20:45:41.526326821 +0000 UTC m=+3440.043099403" Nov 02 20:45:44.748661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.748625 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v"] Nov 02 20:45:44.753226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.753200 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:44.804222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.804183 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v"] Nov 02 20:45:44.846266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.846228 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8d5148\"" Nov 02 20:45:44.846490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.846449 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8d5148-dockercfg-wcbm2\"" Nov 02 20:45:44.861584 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.861540 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9169b8b7-3395-4ec5-92b8-dcdef846353f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:44.861854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.861839 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9tz\" (UniqueName: \"kubernetes.io/projected/9169b8b7-3395-4ec5-92b8-dcdef846353f-kube-api-access-6s9tz\") pod \"payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:44.963383 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.963325 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9169b8b7-3395-4ec5-92b8-dcdef846353f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:44.963674 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.963471 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9tz\" (UniqueName: \"kubernetes.io/projected/9169b8b7-3395-4ec5-92b8-dcdef846353f-kube-api-access-6s9tz\") pod \"payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:44.964070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:44.964051 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9169b8b7-3395-4ec5-92b8-dcdef846353f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:45.016884 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:45.016856 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9tz\" (UniqueName: \"kubernetes.io/projected/9169b8b7-3395-4ec5-92b8-dcdef846353f-kube-api-access-6s9tz\") pod \"payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:45.068751 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:45.068708 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:45.440396 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:45:45.440368 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9169b8b7_3395_4ec5_92b8_dcdef846353f.slice/crio-d74924beba77022d3c816eedf0049ef823cc34f9002bf69b1892b04356b734dd WatchSource:0}: Error finding container d74924beba77022d3c816eedf0049ef823cc34f9002bf69b1892b04356b734dd: Status 404 returned error can't find the container with id d74924beba77022d3c816eedf0049ef823cc34f9002bf69b1892b04356b734dd Nov 02 20:45:45.440396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:45.440390 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v"] Nov 02 20:45:46.353578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:46.353543 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" event={"ID":"9169b8b7-3395-4ec5-92b8-dcdef846353f","Type":"ContainerStarted","Data":"eb19c6649098a2141e6a5d77fe1b0443dc5a2fbf736e45397877e06aec65131a"} Nov 02 20:45:46.354005 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:46.353598 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" event={"ID":"9169b8b7-3395-4ec5-92b8-dcdef846353f","Type":"ContainerStarted","Data":"d74924beba77022d3c816eedf0049ef823cc34f9002bf69b1892b04356b734dd"} Nov 02 20:45:46.354005 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:46.353763 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:45:46.460612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:46.460548 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" podStartSLOduration=2.460529263 podStartE2EDuration="2.460529263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:45:46.458419534 +0000 UTC m=+3444.975192118" watchObservedRunningTime="2026-11-02 20:45:46.460529263 +0000 UTC m=+3444.977301854" Nov 02 20:45:52.322885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:52.322850 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:45:52.323340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:52.322927 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:45:52.698479 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:45:52.698397 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:45:57.370306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:45:57.365467 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:46:02.130970 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.130931 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr"] Nov 02 20:46:02.136321 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.136293 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.185491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.185449 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr"] Nov 02 20:46:02.227227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.227190 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:46:02.227227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.227208 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:46:02.227227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.227223 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:46:02.232185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.232162 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:46:02.286449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.286416 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.286449 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.286451 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkltp\" (UniqueName: \"kubernetes.io/projected/7841753e-6040-44af-b136-73d36c5b9c69-kube-api-access-zkltp\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.286729 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.286641 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.387920 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.387827 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.387920 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.387872 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkltp\" (UniqueName: \"kubernetes.io/projected/7841753e-6040-44af-b136-73d36c5b9c69-kube-api-access-zkltp\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.388120 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.387942 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.388737 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.388710 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.390457 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.390431 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.441749 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.441717 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkltp\" (UniqueName: \"kubernetes.io/projected/7841753e-6040-44af-b136-73d36c5b9c69-kube-api-access-zkltp\") pod \"maas-api-revoke-sub-7b381a8f-qc4tr\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.450834 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.450799 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:02.846777 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.846744 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr"] Nov 02 20:46:02.847122 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:46:02.847095 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7841753e_6040_44af_b136_73d36c5b9c69.slice/crio-6d87ebd2a6403c035c4afa326bcda741d061844c319650e8f09471ec8026b304 WatchSource:0}: Error finding container 6d87ebd2a6403c035c4afa326bcda741d061844c319650e8f09471ec8026b304: Status 404 returned error can't find the container with id 6d87ebd2a6403c035c4afa326bcda741d061844c319650e8f09471ec8026b304 Nov 02 20:46:02.870275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.870242 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn"] Nov 02 20:46:02.876116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.876098 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:02.948041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.948005 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn"] Nov 02 20:46:02.994730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.994695 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8qx\" (UniqueName: \"kubernetes.io/projected/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-kube-api-access-pg8qx\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:02.994902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.994839 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:02.994947 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:02.994918 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.096521 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.096428 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8qx\" (UniqueName: \"kubernetes.io/projected/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-kube-api-access-pg8qx\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.096688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.096558 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.096688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.096629 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.097237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.097211 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.099002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.098983 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.150241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.150209 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8qx\" (UniqueName: \"kubernetes.io/projected/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-kube-api-access-pg8qx\") pod \"maas-api-revoke-sub-96e400b3-xjxkn\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.195349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.195323 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:03.486080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.485983 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" event={"ID":"7841753e-6040-44af-b136-73d36c5b9c69","Type":"ContainerStarted","Data":"383abece7e9980e5a01f9fbabc0a916e9f38d0717940c59a46c4143b791ae408"} Nov 02 20:46:03.486080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.486045 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" event={"ID":"7841753e-6040-44af-b136-73d36c5b9c69","Type":"ContainerStarted","Data":"6d87ebd2a6403c035c4afa326bcda741d061844c319650e8f09471ec8026b304"} Nov 02 20:46:03.565631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.565592 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn"] Nov 02 20:46:03.567212 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:46:03.567182 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cf8c34_b36b_4cd3_8105_c41f93f7a672.slice/crio-5c7b884f853edac986688b07be06fdda757750552d9f32d20a1d8ad9976472e7 WatchSource:0}: Error finding container 5c7b884f853edac986688b07be06fdda757750552d9f32d20a1d8ad9976472e7: Status 404 returned error can't find the container with id 5c7b884f853edac986688b07be06fdda757750552d9f32d20a1d8ad9976472e7 Nov 02 20:46:03.595439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:03.595375 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" podStartSLOduration=1.5953549009999999 podStartE2EDuration="1.595354901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:03.590742296 +0000 UTC m=+3462.107514879" watchObservedRunningTime="2026-11-02 20:46:03.595354901 +0000 UTC m=+3462.112127486" Nov 02 20:46:04.494357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:04.494316 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" event={"ID":"d4cf8c34-b36b-4cd3-8105-c41f93f7a672","Type":"ContainerStarted","Data":"32864fd8396b8025db90ebcdc0ff8317322fb651b6a4f6bdf6bb035f3b3fd4e9"} Nov 02 20:46:04.494858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:04.494410 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" event={"ID":"d4cf8c34-b36b-4cd3-8105-c41f93f7a672","Type":"ContainerStarted","Data":"5c7b884f853edac986688b07be06fdda757750552d9f32d20a1d8ad9976472e7"} Nov 02 20:46:04.496273 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:04.496245 5800 generic.go:356] "Generic (PLEG): container finished" podID="7841753e-6040-44af-b136-73d36c5b9c69" containerID="383abece7e9980e5a01f9fbabc0a916e9f38d0717940c59a46c4143b791ae408" exitCode=0 Nov 02 20:46:04.496428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:04.496330 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" event={"ID":"7841753e-6040-44af-b136-73d36c5b9c69","Type":"ContainerDied","Data":"383abece7e9980e5a01f9fbabc0a916e9f38d0717940c59a46c4143b791ae408"} Nov 02 20:46:04.755459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:04.755394 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" podStartSLOduration=2.755376267 podStartE2EDuration="2.755376267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:04.647970223 +0000 UTC m=+3463.164742809" watchObservedRunningTime="2026-11-02 20:46:04.755376267 +0000 UTC m=+3463.272148853" Nov 02 20:46:05.506748 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.506717 5800 generic.go:356] "Generic (PLEG): container finished" podID="d4cf8c34-b36b-4cd3-8105-c41f93f7a672" containerID="32864fd8396b8025db90ebcdc0ff8317322fb651b6a4f6bdf6bb035f3b3fd4e9" exitCode=0 Nov 02 20:46:05.507182 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.506789 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" event={"ID":"d4cf8c34-b36b-4cd3-8105-c41f93f7a672","Type":"ContainerDied","Data":"32864fd8396b8025db90ebcdc0ff8317322fb651b6a4f6bdf6bb035f3b3fd4e9"} Nov 02 20:46:05.665703 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.662970 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt"] Nov 02 20:46:05.669855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.669832 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.686220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.686192 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:05.730507 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.729069 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt"] Nov 02 20:46:05.831393 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.831256 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7841753e-6040-44af-b136-73d36c5b9c69-kube-api-access-zkltp\" (UniqueName: \"kubernetes.io/projected/7841753e-6040-44af-b136-73d36c5b9c69-kube-api-access-zkltp\") pod \"7841753e-6040-44af-b136-73d36c5b9c69\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " Nov 02 20:46:05.831587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.831428 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-xks-ca\") pod \"7841753e-6040-44af-b136-73d36c5b9c69\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " Nov 02 20:46:05.831587 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.831485 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-openshift-ca\") pod \"7841753e-6040-44af-b136-73d36c5b9c69\" (UID: \"7841753e-6040-44af-b136-73d36c5b9c69\") " Nov 02 20:46:05.831873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.831848 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.832073 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.832051 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.832180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.832086 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4l64\" (UniqueName: \"kubernetes.io/projected/2b770812-57da-48ea-b100-9fc54a7c181f-kube-api-access-k4l64\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.832180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.831918 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-openshift-ca" pod "7841753e-6040-44af-b136-73d36c5b9c69" (UID: "7841753e-6040-44af-b136-73d36c5b9c69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:05.832342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.832271 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:05.833586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.833553 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7841753e-6040-44af-b136-73d36c5b9c69-kube-api-access-zkltp" pod "7841753e-6040-44af-b136-73d36c5b9c69" (UID: "7841753e-6040-44af-b136-73d36c5b9c69"). InnerVolumeSpecName "kube-api-access-zkltp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:05.833689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.833624 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-xks-ca" pod "7841753e-6040-44af-b136-73d36c5b9c69" (UID: "7841753e-6040-44af-b136-73d36c5b9c69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:05.932939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.932902 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.932939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.932943 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.933192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.932961 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4l64\" (UniqueName: \"kubernetes.io/projected/2b770812-57da-48ea-b100-9fc54a7c181f-kube-api-access-k4l64\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.933192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.933108 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkltp\" (UniqueName: \"kubernetes.io/projected/7841753e-6040-44af-b136-73d36c5b9c69-kube-api-access-zkltp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:05.933192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.933120 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7841753e-6040-44af-b136-73d36c5b9c69-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:05.933734 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.933709 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.935458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.935429 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.986696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.986664 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4l64\" (UniqueName: \"kubernetes.io/projected/2b770812-57da-48ea-b100-9fc54a7c181f-kube-api-access-k4l64\") pod \"maas-api-revoke-sub-7a202f3e-5ktbt\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:05.996536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:05.996508 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:06.369042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.369004 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt"] Nov 02 20:46:06.369653 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:46:06.369623 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b770812_57da_48ea_b100_9fc54a7c181f.slice/crio-083be3ea736582754d6f4711aa917e1303ff2944076193c378e9996726d0b7f8 WatchSource:0}: Error finding container 083be3ea736582754d6f4711aa917e1303ff2944076193c378e9996726d0b7f8: Status 404 returned error can't find the container with id 083be3ea736582754d6f4711aa917e1303ff2944076193c378e9996726d0b7f8 Nov 02 20:46:06.514417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.514387 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" Nov 02 20:46:06.514417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.514413 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr" event={"ID":"7841753e-6040-44af-b136-73d36c5b9c69","Type":"ContainerDied","Data":"6d87ebd2a6403c035c4afa326bcda741d061844c319650e8f09471ec8026b304"} Nov 02 20:46:06.514948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.514461 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d87ebd2a6403c035c4afa326bcda741d061844c319650e8f09471ec8026b304" Nov 02 20:46:06.515891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.515867 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" event={"ID":"2b770812-57da-48ea-b100-9fc54a7c181f","Type":"ContainerStarted","Data":"083be3ea736582754d6f4711aa917e1303ff2944076193c378e9996726d0b7f8"} Nov 02 20:46:06.703723 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.703693 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:06.844808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.844773 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-xks-ca\") pod \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " Nov 02 20:46:06.844996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.844868 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-kube-api-access-pg8qx\" (UniqueName: \"kubernetes.io/projected/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-kube-api-access-pg8qx\") pod \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " Nov 02 20:46:06.844996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.844959 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-openshift-ca\") pod \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\" (UID: \"d4cf8c34-b36b-4cd3-8105-c41f93f7a672\") " Nov 02 20:46:06.845902 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.845857 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-openshift-ca" pod "d4cf8c34-b36b-4cd3-8105-c41f93f7a672" (UID: "d4cf8c34-b36b-4cd3-8105-c41f93f7a672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:06.848512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.848463 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-xks-ca" pod "d4cf8c34-b36b-4cd3-8105-c41f93f7a672" (UID: "d4cf8c34-b36b-4cd3-8105-c41f93f7a672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:06.848628 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.848502 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-kube-api-access-pg8qx" pod "d4cf8c34-b36b-4cd3-8105-c41f93f7a672" (UID: "d4cf8c34-b36b-4cd3-8105-c41f93f7a672"). InnerVolumeSpecName "kube-api-access-pg8qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:06.947233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.947136 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8qx\" (UniqueName: \"kubernetes.io/projected/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-kube-api-access-pg8qx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:06.947233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.947172 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:06.947233 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:06.947186 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4cf8c34-b36b-4cd3-8105-c41f93f7a672-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:07.538544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:07.538438 5800 generic.go:356] "Generic (PLEG): container finished" podID="2b770812-57da-48ea-b100-9fc54a7c181f" containerID="42bf7038f11ecfe3197786a952a8ae2b06aad5b4e9c4259be2e833ef66ad60f2" exitCode=0 Nov 02 20:46:07.538544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:07.538524 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" event={"ID":"2b770812-57da-48ea-b100-9fc54a7c181f","Type":"ContainerDied","Data":"42bf7038f11ecfe3197786a952a8ae2b06aad5b4e9c4259be2e833ef66ad60f2"} Nov 02 20:46:07.540484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:07.540453 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" event={"ID":"d4cf8c34-b36b-4cd3-8105-c41f93f7a672","Type":"ContainerDied","Data":"5c7b884f853edac986688b07be06fdda757750552d9f32d20a1d8ad9976472e7"} Nov 02 20:46:07.540626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:07.540504 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c7b884f853edac986688b07be06fdda757750552d9f32d20a1d8ad9976472e7" Nov 02 20:46:07.540626 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:07.540461 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn" Nov 02 20:46:08.720979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.720954 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:08.881735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.881640 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b770812-57da-48ea-b100-9fc54a7c181f-kube-api-access-k4l64\" (UniqueName: \"kubernetes.io/projected/2b770812-57da-48ea-b100-9fc54a7c181f-kube-api-access-k4l64\") pod \"2b770812-57da-48ea-b100-9fc54a7c181f\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " Nov 02 20:46:08.882017 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.881982 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-xks-ca\") pod \"2b770812-57da-48ea-b100-9fc54a7c181f\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " Nov 02 20:46:08.882323 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.882257 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-openshift-ca\") pod \"2b770812-57da-48ea-b100-9fc54a7c181f\" (UID: \"2b770812-57da-48ea-b100-9fc54a7c181f\") " Nov 02 20:46:08.882907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.882881 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-openshift-ca" pod "2b770812-57da-48ea-b100-9fc54a7c181f" (UID: "2b770812-57da-48ea-b100-9fc54a7c181f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:08.886335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.886263 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-xks-ca" pod "2b770812-57da-48ea-b100-9fc54a7c181f" (UID: "2b770812-57da-48ea-b100-9fc54a7c181f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:08.887328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.887303 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b770812-57da-48ea-b100-9fc54a7c181f-kube-api-access-k4l64" pod "2b770812-57da-48ea-b100-9fc54a7c181f" (UID: "2b770812-57da-48ea-b100-9fc54a7c181f"). InnerVolumeSpecName "kube-api-access-k4l64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:08.983681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.983642 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:08.983681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.983678 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4l64\" (UniqueName: \"kubernetes.io/projected/2b770812-57da-48ea-b100-9fc54a7c181f-kube-api-access-k4l64\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:08.983681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:08.983689 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b770812-57da-48ea-b100-9fc54a7c181f-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:09.557003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:09.556973 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" Nov 02 20:46:09.557191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:09.557020 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt" event={"ID":"2b770812-57da-48ea-b100-9fc54a7c181f","Type":"ContainerDied","Data":"083be3ea736582754d6f4711aa917e1303ff2944076193c378e9996726d0b7f8"} Nov 02 20:46:09.557191 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:09.557072 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="083be3ea736582754d6f4711aa917e1303ff2944076193c378e9996726d0b7f8" Nov 02 20:46:12.883907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.883870 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4"] Nov 02 20:46:12.884548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884532 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d4cf8c34-b36b-4cd3-8105-c41f93f7a672" containerName="revoke-keys" Nov 02 20:46:12.884548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884549 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4cf8c34-b36b-4cd3-8105-c41f93f7a672" containerName="revoke-keys" Nov 02 20:46:12.884638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884573 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2b770812-57da-48ea-b100-9fc54a7c181f" containerName="revoke-keys" Nov 02 20:46:12.884638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884578 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b770812-57da-48ea-b100-9fc54a7c181f" containerName="revoke-keys" Nov 02 20:46:12.884638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884587 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7841753e-6040-44af-b136-73d36c5b9c69" containerName="revoke-keys" Nov 02 20:46:12.884638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884593 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7841753e-6040-44af-b136-73d36c5b9c69" containerName="revoke-keys" Nov 02 20:46:12.884772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884683 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4cf8c34-b36b-4cd3-8105-c41f93f7a672" containerName="revoke-keys" Nov 02 20:46:12.884772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884694 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b770812-57da-48ea-b100-9fc54a7c181f" containerName="revoke-keys" Nov 02 20:46:12.884772 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.884700 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7841753e-6040-44af-b136-73d36c5b9c69" containerName="revoke-keys" Nov 02 20:46:12.889456 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.889439 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:12.931353 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.931326 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:12.931511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.931491 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:12.931559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.931532 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqk7r\" (UniqueName: \"kubernetes.io/projected/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-kube-api-access-vqk7r\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:12.940624 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.940595 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4"] Nov 02 20:46:12.982296 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.982203 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:46:12.982481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.982383 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:46:12.982481 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.982422 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:46:12.984388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:12.984375 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:46:13.032691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.032650 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk7r\" (UniqueName: \"kubernetes.io/projected/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-kube-api-access-vqk7r\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:13.032893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.032765 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:13.032983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.032919 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:13.033454 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.033427 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:13.035308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.035271 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:13.085682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.085648 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk7r\" (UniqueName: \"kubernetes.io/projected/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-kube-api-access-vqk7r\") pod \"maas-api-revoke-sub-12234760-k75c4\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:13.204726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.204638 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:13.575596 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.575557 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4"] Nov 02 20:46:13.577970 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:46:13.577945 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c57e2f_6f21_4a3c_87d2_2df57937c4c7.slice/crio-e4967796996cd6dd2f5b1acafecdd374044fe41811b5c59c4771eb1a2643e69d WatchSource:0}: Error finding container e4967796996cd6dd2f5b1acafecdd374044fe41811b5c59c4771eb1a2643e69d: Status 404 returned error can't find the container with id e4967796996cd6dd2f5b1acafecdd374044fe41811b5c59c4771eb1a2643e69d Nov 02 20:46:13.587325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.587266 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" event={"ID":"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7","Type":"ContainerStarted","Data":"e4967796996cd6dd2f5b1acafecdd374044fe41811b5c59c4771eb1a2643e69d"} Nov 02 20:46:13.634242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.634215 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68"] Nov 02 20:46:13.639857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.639831 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.691297 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.691248 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68"] Nov 02 20:46:13.742320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.740923 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.742320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.741175 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.742320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.741210 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t59w\" (UniqueName: \"kubernetes.io/projected/06636ce0-2746-4dfd-9cff-09252b1b3a18-kube-api-access-9t59w\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.842632 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.842597 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.842802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.842640 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t59w\" (UniqueName: \"kubernetes.io/projected/06636ce0-2746-4dfd-9cff-09252b1b3a18-kube-api-access-9t59w\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.842802 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.842681 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.843439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.843412 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.845026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.845003 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.895599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.895569 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t59w\" (UniqueName: \"kubernetes.io/projected/06636ce0-2746-4dfd-9cff-09252b1b3a18-kube-api-access-9t59w\") pod \"maas-api-revoke-sub-e265790a-fzv68\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:13.959766 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:13.959740 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:14.330337 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:14.330262 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68"] Nov 02 20:46:14.330925 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:46:14.330891 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06636ce0_2746_4dfd_9cff_09252b1b3a18.slice/crio-fda83767f6e2efd5b5dff0a289e74384c25f3caa39c2674456ec65b1087e96d1 WatchSource:0}: Error finding container fda83767f6e2efd5b5dff0a289e74384c25f3caa39c2674456ec65b1087e96d1: Status 404 returned error can't find the container with id fda83767f6e2efd5b5dff0a289e74384c25f3caa39c2674456ec65b1087e96d1 Nov 02 20:46:14.596971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:14.596920 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" event={"ID":"06636ce0-2746-4dfd-9cff-09252b1b3a18","Type":"ContainerStarted","Data":"fda83767f6e2efd5b5dff0a289e74384c25f3caa39c2674456ec65b1087e96d1"} Nov 02 20:46:14.598948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:14.598921 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" event={"ID":"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7","Type":"ContainerStarted","Data":"8072bbb1850685bc143da23bfbafcd26f010ee7def31f0069dc5f87b013da13c"} Nov 02 20:46:14.709301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:14.709238 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" podStartSLOduration=2.7092234299999998 podStartE2EDuration="2.70922343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:14.704747499 +0000 UTC m=+3473.221520083" watchObservedRunningTime="2026-11-02 20:46:14.70922343 +0000 UTC m=+3473.225996015" Nov 02 20:46:15.607831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:15.607793 5800 generic.go:356] "Generic (PLEG): container finished" podID="06636ce0-2746-4dfd-9cff-09252b1b3a18" containerID="5d215b9386af05a8f7b4d7f32fc768ba4e9d59b08b4a20e557b43fa8dc92ae1b" exitCode=0 Nov 02 20:46:15.608317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:15.607884 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" event={"ID":"06636ce0-2746-4dfd-9cff-09252b1b3a18","Type":"ContainerDied","Data":"5d215b9386af05a8f7b4d7f32fc768ba4e9d59b08b4a20e557b43fa8dc92ae1b"} Nov 02 20:46:15.609644 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:15.609612 5800 generic.go:356] "Generic (PLEG): container finished" podID="c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" containerID="8072bbb1850685bc143da23bfbafcd26f010ee7def31f0069dc5f87b013da13c" exitCode=0 Nov 02 20:46:15.609731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:15.609639 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" event={"ID":"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7","Type":"ContainerDied","Data":"8072bbb1850685bc143da23bfbafcd26f010ee7def31f0069dc5f87b013da13c"} Nov 02 20:46:16.464660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.464629 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc"] Nov 02 20:46:16.469034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.469015 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.520354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.520318 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc"] Nov 02 20:46:16.572894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.572852 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.573076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.572963 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97jb\" (UniqueName: \"kubernetes.io/projected/000ccce9-467a-4c57-a38c-f5116dedf5f8-kube-api-access-m97jb\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.573076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.573045 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.674728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.674694 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.675136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.674755 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97jb\" (UniqueName: \"kubernetes.io/projected/000ccce9-467a-4c57-a38c-f5116dedf5f8-kube-api-access-m97jb\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.675136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.674799 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.675550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.675493 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.677495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.677469 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.736349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.736246 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97jb\" (UniqueName: \"kubernetes.io/projected/000ccce9-467a-4c57-a38c-f5116dedf5f8-kube-api-access-m97jb\") pod \"maas-api-revoke-sub-1aa9ee61-6lbkc\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.783261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.783228 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:16.803520 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.803492 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:16.825570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.825545 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:16.877618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.877586 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-xks-ca\") pod \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " Nov 02 20:46:16.877785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.877644 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-kube-api-access-vqk7r\" (UniqueName: \"kubernetes.io/projected/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-kube-api-access-vqk7r\") pod \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " Nov 02 20:46:16.877785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.877680 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-openshift-ca\") pod \"06636ce0-2746-4dfd-9cff-09252b1b3a18\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " Nov 02 20:46:16.877785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.877714 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-xks-ca\") pod \"06636ce0-2746-4dfd-9cff-09252b1b3a18\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " Nov 02 20:46:16.877785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.877759 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06636ce0-2746-4dfd-9cff-09252b1b3a18-kube-api-access-9t59w\" (UniqueName: \"kubernetes.io/projected/06636ce0-2746-4dfd-9cff-09252b1b3a18-kube-api-access-9t59w\") pod \"06636ce0-2746-4dfd-9cff-09252b1b3a18\" (UID: \"06636ce0-2746-4dfd-9cff-09252b1b3a18\") " Nov 02 20:46:16.877785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.877786 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-openshift-ca\") pod \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\" (UID: \"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7\") " Nov 02 20:46:16.878129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.878100 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-openshift-ca" pod "06636ce0-2746-4dfd-9cff-09252b1b3a18" (UID: "06636ce0-2746-4dfd-9cff-09252b1b3a18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:16.878377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.878349 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.878502 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.878471 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-openshift-ca" pod "c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" (UID: "c1c57e2f-6f21-4a3c-87d2-2df57937c4c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:16.880151 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.880126 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06636ce0-2746-4dfd-9cff-09252b1b3a18-kube-api-access-9t59w" pod "06636ce0-2746-4dfd-9cff-09252b1b3a18" (UID: "06636ce0-2746-4dfd-9cff-09252b1b3a18"). InnerVolumeSpecName "kube-api-access-9t59w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:16.881236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.881215 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-xks-ca" pod "06636ce0-2746-4dfd-9cff-09252b1b3a18" (UID: "06636ce0-2746-4dfd-9cff-09252b1b3a18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:16.881462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.881345 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-xks-ca" pod "c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" (UID: "c1c57e2f-6f21-4a3c-87d2-2df57937c4c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:16.881462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.881353 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-kube-api-access-vqk7r" pod "c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" (UID: "c1c57e2f-6f21-4a3c-87d2-2df57937c4c7"). InnerVolumeSpecName "kube-api-access-vqk7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:16.979181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.979137 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06636ce0-2746-4dfd-9cff-09252b1b3a18-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.979181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.979171 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t59w\" (UniqueName: \"kubernetes.io/projected/06636ce0-2746-4dfd-9cff-09252b1b3a18-kube-api-access-9t59w\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.979181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.979186 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.979475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.979199 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.979475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:16.979211 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqk7r\" (UniqueName: \"kubernetes.io/projected/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7-kube-api-access-vqk7r\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:17.169174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.169142 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc"] Nov 02 20:46:17.169376 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:46:17.169154 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000ccce9_467a_4c57_a38c_f5116dedf5f8.slice/crio-df0f0f981aaea4becc4c2d875ef086d5f7f7cf32772b0b1635fc272417282207 WatchSource:0}: Error finding container df0f0f981aaea4becc4c2d875ef086d5f7f7cf32772b0b1635fc272417282207: Status 404 returned error can't find the container with id df0f0f981aaea4becc4c2d875ef086d5f7f7cf32772b0b1635fc272417282207 Nov 02 20:46:17.626392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.626363 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" Nov 02 20:46:17.626602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.626401 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4" event={"ID":"c1c57e2f-6f21-4a3c-87d2-2df57937c4c7","Type":"ContainerDied","Data":"e4967796996cd6dd2f5b1acafecdd374044fe41811b5c59c4771eb1a2643e69d"} Nov 02 20:46:17.626602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.626447 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4967796996cd6dd2f5b1acafecdd374044fe41811b5c59c4771eb1a2643e69d" Nov 02 20:46:17.628446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.628415 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" event={"ID":"000ccce9-467a-4c57-a38c-f5116dedf5f8","Type":"ContainerStarted","Data":"ea664614dcd5aa02eb11578b07ec5303ed381d01a1a15c8ff4494da59287d203"} Nov 02 20:46:17.628583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.628473 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" event={"ID":"000ccce9-467a-4c57-a38c-f5116dedf5f8","Type":"ContainerStarted","Data":"df0f0f981aaea4becc4c2d875ef086d5f7f7cf32772b0b1635fc272417282207"} Nov 02 20:46:17.630452 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.630427 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" event={"ID":"06636ce0-2746-4dfd-9cff-09252b1b3a18","Type":"ContainerDied","Data":"fda83767f6e2efd5b5dff0a289e74384c25f3caa39c2674456ec65b1087e96d1"} Nov 02 20:46:17.630548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.630471 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fda83767f6e2efd5b5dff0a289e74384c25f3caa39c2674456ec65b1087e96d1" Nov 02 20:46:17.630548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.630486 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68" Nov 02 20:46:17.737047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:17.736987 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" podStartSLOduration=1.7369723289999999 podStartE2EDuration="1.736972329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:17.733113339 +0000 UTC m=+3476.249885924" watchObservedRunningTime="2026-11-02 20:46:17.736972329 +0000 UTC m=+3476.253744948" Nov 02 20:46:18.639590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:18.639556 5800 generic.go:356] "Generic (PLEG): container finished" podID="000ccce9-467a-4c57-a38c-f5116dedf5f8" containerID="ea664614dcd5aa02eb11578b07ec5303ed381d01a1a15c8ff4494da59287d203" exitCode=0 Nov 02 20:46:18.639793 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:18.639650 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" event={"ID":"000ccce9-467a-4c57-a38c-f5116dedf5f8","Type":"ContainerDied","Data":"ea664614dcd5aa02eb11578b07ec5303ed381d01a1a15c8ff4494da59287d203"} Nov 02 20:46:19.808003 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.807977 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:19.914184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.914083 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-openshift-ca\") pod \"000ccce9-467a-4c57-a38c-f5116dedf5f8\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " Nov 02 20:46:19.914184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.914157 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000ccce9-467a-4c57-a38c-f5116dedf5f8-kube-api-access-m97jb\" (UniqueName: \"kubernetes.io/projected/000ccce9-467a-4c57-a38c-f5116dedf5f8-kube-api-access-m97jb\") pod \"000ccce9-467a-4c57-a38c-f5116dedf5f8\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " Nov 02 20:46:19.914476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.914275 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-xks-ca\") pod \"000ccce9-467a-4c57-a38c-f5116dedf5f8\" (UID: \"000ccce9-467a-4c57-a38c-f5116dedf5f8\") " Nov 02 20:46:19.914571 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.914541 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-openshift-ca" pod "000ccce9-467a-4c57-a38c-f5116dedf5f8" (UID: "000ccce9-467a-4c57-a38c-f5116dedf5f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:19.914865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.914837 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:19.916695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.916661 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-xks-ca" pod "000ccce9-467a-4c57-a38c-f5116dedf5f8" (UID: "000ccce9-467a-4c57-a38c-f5116dedf5f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:19.916809 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:19.916687 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000ccce9-467a-4c57-a38c-f5116dedf5f8-kube-api-access-m97jb" pod "000ccce9-467a-4c57-a38c-f5116dedf5f8" (UID: "000ccce9-467a-4c57-a38c-f5116dedf5f8"). InnerVolumeSpecName "kube-api-access-m97jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:20.016546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:20.016511 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97jb\" (UniqueName: \"kubernetes.io/projected/000ccce9-467a-4c57-a38c-f5116dedf5f8-kube-api-access-m97jb\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:20.016546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:20.016546 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000ccce9-467a-4c57-a38c-f5116dedf5f8-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:20.659583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:20.659553 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" Nov 02 20:46:20.659583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:20.659572 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc" event={"ID":"000ccce9-467a-4c57-a38c-f5116dedf5f8","Type":"ContainerDied","Data":"df0f0f981aaea4becc4c2d875ef086d5f7f7cf32772b0b1635fc272417282207"} Nov 02 20:46:20.659864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:20.659631 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df0f0f981aaea4becc4c2d875ef086d5f7f7cf32772b0b1635fc272417282207" Nov 02 20:46:22.698721 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:46:22.698682 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:46:31.014534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:31.014499 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z"] Nov 02 20:46:46.035873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.035832 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" podUID="772652a0-a403-42fe-a14a-adb11615b9ec" containerName="main" containerID="cri-o://c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92" gracePeriod=45 Nov 02 20:46:46.319690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.319664 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.441598 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-model-cache\") pod \"772652a0-a403-42fe-a14a-adb11615b9ec\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.441731 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-dshm\") pod \"772652a0-a403-42fe-a14a-adb11615b9ec\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.441800 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-home\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-home\") pod \"772652a0-a403-42fe-a14a-adb11615b9ec\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.441849 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772652a0-a403-42fe-a14a-adb11615b9ec-kube-api-access-rtwm6\" (UniqueName: \"kubernetes.io/projected/772652a0-a403-42fe-a14a-adb11615b9ec-kube-api-access-rtwm6\") pod \"772652a0-a403-42fe-a14a-adb11615b9ec\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.441850 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-model-cache" pod "772652a0-a403-42fe-a14a-adb11615b9ec" (UID: "772652a0-a403-42fe-a14a-adb11615b9ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.441918 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-tmp-dir\") pod \"772652a0-a403-42fe-a14a-adb11615b9ec\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.441965 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772652a0-a403-42fe-a14a-adb11615b9ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/772652a0-a403-42fe-a14a-adb11615b9ec-tls-certs\") pod \"772652a0-a403-42fe-a14a-adb11615b9ec\" (UID: \"772652a0-a403-42fe-a14a-adb11615b9ec\") " Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.442053 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-home" pod "772652a0-a403-42fe-a14a-adb11615b9ec" (UID: "772652a0-a403-42fe-a14a-adb11615b9ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.442260 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-tmp-dir" pod "772652a0-a403-42fe-a14a-adb11615b9ec" (UID: "772652a0-a403-42fe-a14a-adb11615b9ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.442513 5800 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-model-cache\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.442537 5800 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-home\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:46.442578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.442550 5800 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-tmp-dir\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:46.444843 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.444804 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772652a0-a403-42fe-a14a-adb11615b9ec-tls-certs" pod "772652a0-a403-42fe-a14a-adb11615b9ec" (UID: "772652a0-a403-42fe-a14a-adb11615b9ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:46.444986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.444939 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-dshm" pod "772652a0-a403-42fe-a14a-adb11615b9ec" (UID: "772652a0-a403-42fe-a14a-adb11615b9ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:46.444986 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.444958 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772652a0-a403-42fe-a14a-adb11615b9ec-kube-api-access-rtwm6" pod "772652a0-a403-42fe-a14a-adb11615b9ec" (UID: "772652a0-a403-42fe-a14a-adb11615b9ec"). InnerVolumeSpecName "kube-api-access-rtwm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:46.544051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.544016 5800 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772652a0-a403-42fe-a14a-adb11615b9ec-dshm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:46.544051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.544048 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtwm6\" (UniqueName: \"kubernetes.io/projected/772652a0-a403-42fe-a14a-adb11615b9ec-kube-api-access-rtwm6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:46.544272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.544064 5800 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772652a0-a403-42fe-a14a-adb11615b9ec-tls-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:46.869361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.869325 5800 generic.go:356] "Generic (PLEG): container finished" podID="772652a0-a403-42fe-a14a-adb11615b9ec" containerID="c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92" exitCode=0 Nov 02 20:46:46.869544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.869368 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" Nov 02 20:46:46.869544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.869369 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" event={"ID":"772652a0-a403-42fe-a14a-adb11615b9ec","Type":"ContainerDied","Data":"c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92"} Nov 02 20:46:46.869544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.869481 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z" event={"ID":"772652a0-a403-42fe-a14a-adb11615b9ec","Type":"ContainerDied","Data":"bb5da0c398c718c29e0b6d094adb80f4accd028efe1cd7d45ad8a8b0f95d3601"} Nov 02 20:46:46.869544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.869504 5800 scope.go:122] "RemoveContainer" containerID="c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92" Nov 02 20:46:46.880719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.880703 5800 scope.go:122] "RemoveContainer" containerID="c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92" Nov 02 20:46:46.880986 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:46:46.880970 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92\": container with ID starting with c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92 not found: ID does not exist" containerID="c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92" Nov 02 20:46:46.881041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:46.880992 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92"} err="failed to get container status \"c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92\": rpc error: code = NotFound desc = could not find container \"c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92\": container with ID starting with c23d7a799058ef3cab1f383df2c9b82ec38dd08bc6ee4afc4604c37925b54e92 not found: ID does not exist" Nov 02 20:46:47.023325 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:47.023277 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z"] Nov 02 20:46:47.069645 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:47.069611 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a9ed60/auth-test-model-a9ed60-kserve-7684fc7cb9-wkz7z"] Nov 02 20:46:47.626476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:46:47.626445 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772652a0-a403-42fe-a14a-adb11615b9ec" path="/var/lib/kubelet/pods/772652a0-a403-42fe-a14a-adb11615b9ec/volumes" Nov 02 20:46:52.699901 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:46:52.699857 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:47:02.440029 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.439952 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7"] Nov 02 20:47:02.440705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440685 5800 cpu_manager.go:414] "containerMap: removing container" podUID="772652a0-a403-42fe-a14a-adb11615b9ec" containerName="main" Nov 02 20:47:02.440705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440706 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772652a0-a403-42fe-a14a-adb11615b9ec" containerName="main" Nov 02 20:47:02.440857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440720 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" containerName="revoke-keys" Nov 02 20:47:02.440857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440726 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" containerName="revoke-keys" Nov 02 20:47:02.440857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440753 5800 cpu_manager.go:414] "containerMap: removing container" podUID="06636ce0-2746-4dfd-9cff-09252b1b3a18" containerName="revoke-keys" Nov 02 20:47:02.440857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440759 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06636ce0-2746-4dfd-9cff-09252b1b3a18" containerName="revoke-keys" Nov 02 20:47:02.440857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440769 5800 cpu_manager.go:414] "containerMap: removing container" podUID="000ccce9-467a-4c57-a38c-f5116dedf5f8" containerName="revoke-keys" Nov 02 20:47:02.440857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440773 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000ccce9-467a-4c57-a38c-f5116dedf5f8" containerName="revoke-keys" Nov 02 20:47:02.440857 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440857 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="06636ce0-2746-4dfd-9cff-09252b1b3a18" containerName="revoke-keys" Nov 02 20:47:02.441209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440866 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="000ccce9-467a-4c57-a38c-f5116dedf5f8" containerName="revoke-keys" Nov 02 20:47:02.441209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440875 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="772652a0-a403-42fe-a14a-adb11615b9ec" containerName="main" Nov 02 20:47:02.441209 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.440883 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" containerName="revoke-keys" Nov 02 20:47:02.444705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.444684 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464399 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c502ded-6a75-4933-8d40-aec9c972988c-istio-podinfo\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464438 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-envoy\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464584 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c502ded-6a75-4933-8d40-aec9c972988c-istiod-ca-cert\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464681 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464648 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twn9w\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-kube-api-access-twn9w\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464745 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-credential-socket\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464811 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464788 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-data\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464866 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-istio-token\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.464961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464947 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-certs\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.465045 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.464996 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-socket\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.518645 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.518609 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7"] Nov 02 20:47:02.551916 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.551887 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-720bda-openshift-default-dockercfg-5bx4q\"" Nov 02 20:47:02.566562 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566524 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twn9w\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-kube-api-access-twn9w\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.566774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566585 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-credential-socket\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.566774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566629 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-data\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.566774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566663 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-istio-token\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.566774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566701 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-certs\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.566774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566745 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-socket\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566826 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c502ded-6a75-4933-8d40-aec9c972988c-istio-podinfo\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566857 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-envoy\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.566953 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c502ded-6a75-4933-8d40-aec9c972988c-istiod-ca-cert\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.567036 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-credential-socket\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.567049 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-data\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567719 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.567689 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-socket\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.567832 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-certs\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.567937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.567891 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c502ded-6a75-4933-8d40-aec9c972988c-istiod-ca-cert\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.569516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.569494 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-envoy\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.569842 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.569818 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c502ded-6a75-4933-8d40-aec9c972988c-istio-podinfo\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.623691 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.623653 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twn9w\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-kube-api-access-twn9w\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.624206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.624182 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-istio-token\") pod \"e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:02.764417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:02.762558 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:03.376222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.376120 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp"] Nov 02 20:47:03.380797 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.380768 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.418009 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.417972 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7"] Nov 02 20:47:03.418715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418678 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-data\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.418864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418719 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-envoy\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.418864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418752 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146372b4-37e6-4ca6-976b-5f4cae670a6e-istiod-ca-cert\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.418864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418790 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-credential-socket\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.418864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418817 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5tx\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-kube-api-access-gk5tx\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.418864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418858 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-socket\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.419164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418888 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-podinfo\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.419164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.418933 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-certs\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.419164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.419033 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-token\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.435072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.434961 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:03.435235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.435045 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:03.435367 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.435269 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:03.460650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.460622 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-608c11-openshift-default-dockercfg-jppsm\"" Nov 02 20:47:03.469340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.468548 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp"] Nov 02 20:47:03.520262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520224 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-envoy\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520325 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146372b4-37e6-4ca6-976b-5f4cae670a6e-istiod-ca-cert\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520379 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-credential-socket\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520406 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5tx\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-kube-api-access-gk5tx\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520449 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-socket\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520486 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-podinfo\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520534 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-certs\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520685 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520650 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-token\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.520904 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.520752 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-data\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.521188 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.521166 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-data\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.523235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.521637 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-credential-socket\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.523235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.521683 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-socket\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.523235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.522313 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146372b4-37e6-4ca6-976b-5f4cae670a6e-istiod-ca-cert\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.523235 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.522320 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-certs\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.527856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.527828 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-podinfo\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.528112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.528088 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-envoy\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.587333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.582006 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5tx\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-kube-api-access-gk5tx\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.587333 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.586966 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-token\") pod \"e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:03.699557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:03.699393 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:04.039895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.032645 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" event={"ID":"5c502ded-6a75-4933-8d40-aec9c972988c","Type":"ContainerStarted","Data":"c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f"} Nov 02 20:47:04.039895 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.032760 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" event={"ID":"5c502ded-6a75-4933-8d40-aec9c972988c","Type":"ContainerStarted","Data":"32073ffe44e860373f466805bfd2b6feb402a202729a401d499d9864707264a2"} Nov 02 20:47:04.147786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.147726 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podStartSLOduration=2.147710363 podStartE2EDuration="2.147710363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:04.142450935 +0000 UTC m=+3522.659223545" watchObservedRunningTime="2026-11-02 20:47:04.147710363 +0000 UTC m=+3522.664482997" Nov 02 20:47:04.200010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.199983 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp"] Nov 02 20:47:04.202175 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:47:04.202140 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146372b4_37e6_4ca6_976b_5f4cae670a6e.slice/crio-0b250af6937173103233a883465b30b99dc950b797bc81d93e8370c233b96d5e WatchSource:0}: Error finding container 0b250af6937173103233a883465b30b99dc950b797bc81d93e8370c233b96d5e: Status 404 returned error can't find the container with id 0b250af6937173103233a883465b30b99dc950b797bc81d93e8370c233b96d5e Nov 02 20:47:04.205040 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.205002 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:04.205153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.205074 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:04.205153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.205104 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:47:04.763478 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.763447 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:04.769679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:04.769651 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:05.044032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:05.043938 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" event={"ID":"146372b4-37e6-4ca6-976b-5f4cae670a6e","Type":"ContainerStarted","Data":"75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e"} Nov 02 20:47:05.044032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:05.044007 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" event={"ID":"146372b4-37e6-4ca6-976b-5f4cae670a6e","Type":"ContainerStarted","Data":"0b250af6937173103233a883465b30b99dc950b797bc81d93e8370c233b96d5e"} Nov 02 20:47:05.044435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:05.044402 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:05.045609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:05.045588 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:47:05.155362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:05.155295 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podStartSLOduration=2.155263875 podStartE2EDuration="2.155263875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:05.152595957 +0000 UTC m=+3523.669368576" watchObservedRunningTime="2026-11-02 20:47:05.155263875 +0000 UTC m=+3523.672036459" Nov 02 20:47:05.710469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:05.710428 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:05.718419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:05.718390 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:06.054534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:06.054501 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:06.055364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:06.055342 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:47:12.059785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.059752 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9"] Nov 02 20:47:12.064519 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.064485 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.114582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.114551 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9"] Nov 02 20:47:12.155683 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.155654 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bbdf74-96a0-4d04-a62a-b6907f04cfca-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.155867 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.155714 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw967\" (UniqueName: \"kubernetes.io/projected/58bbdf74-96a0-4d04-a62a-b6907f04cfca-kube-api-access-hw967\") pod \"payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.156597 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.156564 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a7c5d1-dockercfg-nj7sm\"" Nov 02 20:47:12.156597 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.156582 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a7c5d1\"" Nov 02 20:47:12.256718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.256680 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw967\" (UniqueName: \"kubernetes.io/projected/58bbdf74-96a0-4d04-a62a-b6907f04cfca-kube-api-access-hw967\") pod \"payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.256937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.256917 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bbdf74-96a0-4d04-a62a-b6907f04cfca-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.257655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.257632 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bbdf74-96a0-4d04-a62a-b6907f04cfca-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.309657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.309624 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw967\" (UniqueName: \"kubernetes.io/projected/58bbdf74-96a0-4d04-a62a-b6907f04cfca-kube-api-access-hw967\") pod \"payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.381109 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.381021 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:12.770474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:12.770443 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9"] Nov 02 20:47:12.770873 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:47:12.770844 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bbdf74_96a0_4d04_a62a_b6907f04cfca.slice/crio-facd87d85339b1bab03afdaae0f89af3989c695da7bb990bc844f2bdbdad3735 WatchSource:0}: Error finding container facd87d85339b1bab03afdaae0f89af3989c695da7bb990bc844f2bdbdad3735: Status 404 returned error can't find the container with id facd87d85339b1bab03afdaae0f89af3989c695da7bb990bc844f2bdbdad3735 Nov 02 20:47:13.108858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:13.108769 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" event={"ID":"58bbdf74-96a0-4d04-a62a-b6907f04cfca","Type":"ContainerStarted","Data":"2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586"} Nov 02 20:47:13.108858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:13.108833 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" event={"ID":"58bbdf74-96a0-4d04-a62a-b6907f04cfca","Type":"ContainerStarted","Data":"facd87d85339b1bab03afdaae0f89af3989c695da7bb990bc844f2bdbdad3735"} Nov 02 20:47:13.109270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:13.108861 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:13.216652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:13.216597 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" podStartSLOduration=1.216582175 podStartE2EDuration="1.216582175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:47:13.213578424 +0000 UTC m=+3531.730351009" watchObservedRunningTime="2026-11-02 20:47:13.216582175 +0000 UTC m=+3531.733354760" Nov 02 20:47:22.697590 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:47:22.697549 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:47:24.119316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:47:24.119269 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:47:52.696564 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:47:52.696525 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:48:22.309007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:22.308975 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:48:22.324833 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:22.324807 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:48:22.715635 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:48:22.715531 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:48:32.495636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.495548 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb"] Nov 02 20:48:32.499959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.499936 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.574516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.570981 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb"] Nov 02 20:48:32.599303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.598259 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-27e421\"" Nov 02 20:48:32.599303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.598478 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-27e421-dockercfg-k2gnq\"" Nov 02 20:48:32.642305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.639230 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a14cd55-526a-4d37-aaaf-e865adaca4f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.642305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.639355 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxvg\" (UniqueName: \"kubernetes.io/projected/2a14cd55-526a-4d37-aaaf-e865adaca4f4-kube-api-access-9jxvg\") pod \"payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.670431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.670388 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc"] Nov 02 20:48:32.675930 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.675905 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:32.718807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.718772 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc"] Nov 02 20:48:32.741138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.741099 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a14cd55-526a-4d37-aaaf-e865adaca4f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.741395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.741179 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxvg\" (UniqueName: \"kubernetes.io/projected/2a14cd55-526a-4d37-aaaf-e865adaca4f4-kube-api-access-9jxvg\") pod \"payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.741395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.741214 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307b9e36-2241-4431-84de-cc009ad00f35-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:32.741395 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.741365 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjlm\" (UniqueName: \"kubernetes.io/projected/307b9e36-2241-4431-84de-cc009ad00f35-kube-api-access-hcjlm\") pod \"payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:32.743277 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.743256 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a14cd55-526a-4d37-aaaf-e865adaca4f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.796590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.796552 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxvg\" (UniqueName: \"kubernetes.io/projected/2a14cd55-526a-4d37-aaaf-e865adaca4f4-kube-api-access-9jxvg\") pod \"payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.815484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.815458 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:32.842482 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.842448 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307b9e36-2241-4431-84de-cc009ad00f35-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:32.842647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.842612 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjlm\" (UniqueName: \"kubernetes.io/projected/307b9e36-2241-4431-84de-cc009ad00f35-kube-api-access-hcjlm\") pod \"payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:32.843276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.843251 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307b9e36-2241-4431-84de-cc009ad00f35-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:32.898089 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.898018 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjlm\" (UniqueName: \"kubernetes.io/projected/307b9e36-2241-4431-84de-cc009ad00f35-kube-api-access-hcjlm\") pod \"payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:32.990887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:32.990838 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:33.197461 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:48:33.197264 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a14cd55_526a_4d37_aaaf_e865adaca4f4.slice/crio-1e72ed9d049a4e074c0ee6ce3867f21ff48889886c67b38685450b3493ae73b7 WatchSource:0}: Error finding container 1e72ed9d049a4e074c0ee6ce3867f21ff48889886c67b38685450b3493ae73b7: Status 404 returned error can't find the container with id 1e72ed9d049a4e074c0ee6ce3867f21ff48889886c67b38685450b3493ae73b7 Nov 02 20:48:33.197461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.197384 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb"] Nov 02 20:48:33.369216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.369192 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc"] Nov 02 20:48:33.371253 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:48:33.371221 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307b9e36_2241_4431_84de_cc009ad00f35.slice/crio-5ab32f559d140201709d837317a3326d3fc09c7640bcc29a21751f39df10a773 WatchSource:0}: Error finding container 5ab32f559d140201709d837317a3326d3fc09c7640bcc29a21751f39df10a773: Status 404 returned error can't find the container with id 5ab32f559d140201709d837317a3326d3fc09c7640bcc29a21751f39df10a773 Nov 02 20:48:33.865580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.865542 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" event={"ID":"307b9e36-2241-4431-84de-cc009ad00f35","Type":"ContainerStarted","Data":"a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86"} Nov 02 20:48:33.866084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.865636 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" event={"ID":"307b9e36-2241-4431-84de-cc009ad00f35","Type":"ContainerStarted","Data":"5ab32f559d140201709d837317a3326d3fc09c7640bcc29a21751f39df10a773"} Nov 02 20:48:33.866084 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.865962 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:33.867888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.867857 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" event={"ID":"2a14cd55-526a-4d37-aaaf-e865adaca4f4","Type":"ContainerStarted","Data":"ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f"} Nov 02 20:48:33.868023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.867910 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" event={"ID":"2a14cd55-526a-4d37-aaaf-e865adaca4f4","Type":"ContainerStarted","Data":"1e72ed9d049a4e074c0ee6ce3867f21ff48889886c67b38685450b3493ae73b7"} Nov 02 20:48:33.868023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.867956 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:33.972511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:33.972454 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" podStartSLOduration=1.972439077 podStartE2EDuration="1.972439077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:33.970838192 +0000 UTC m=+3612.487610778" watchObservedRunningTime="2026-11-02 20:48:33.972439077 +0000 UTC m=+3612.489211662" Nov 02 20:48:34.077781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:34.077719 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" podStartSLOduration=2.077703511 podStartE2EDuration="2.077703511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:48:34.076156794 +0000 UTC m=+3612.592929381" watchObservedRunningTime="2026-11-02 20:48:34.077703511 +0000 UTC m=+3612.594476096" Nov 02 20:48:44.878564 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:44.878533 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:48:44.879031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:48:44.878613 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:48:52.698299 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:48:52.698248 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:49:15.044878 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:15.044842 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj"] Nov 02 20:49:15.090952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:15.090914 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af1a238e-ndpkj"] Nov 02 20:49:15.626256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:15.625499 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213a09d0-3c6c-4d8f-ba46-c5e430482dd0" path="/var/lib/kubelet/pods/213a09d0-3c6c-4d8f-ba46-c5e430482dd0/volumes" Nov 02 20:49:17.261153 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.261120 5800 generic.go:356] "Generic (PLEG): container finished" podID="5c502ded-6a75-4933-8d40-aec9c972988c" containerID="c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f" exitCode=0 Nov 02 20:49:17.261675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.261192 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" event={"ID":"5c502ded-6a75-4933-8d40-aec9c972988c","Type":"ContainerDied","Data":"c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f"} Nov 02 20:49:17.261733 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.261679 5800 scope.go:122] "RemoveContainer" containerID="c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f" Nov 02 20:49:17.263468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.263440 5800 generic.go:356] "Generic (PLEG): container finished" podID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerID="75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e" exitCode=0 Nov 02 20:49:17.263569 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.263509 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" event={"ID":"146372b4-37e6-4ca6-976b-5f4cae670a6e","Type":"ContainerDied","Data":"75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e"} Nov 02 20:49:17.263845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.263831 5800 scope.go:122] "RemoveContainer" containerID="75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e" Nov 02 20:49:17.263918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.263884 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:49:17.263996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.263979 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:49:17.264047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.264031 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:49:17.265735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.265702 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:49:17.265854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.265781 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:49:17.265854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:17.265829 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:49:18.291780 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.291746 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" event={"ID":"5c502ded-6a75-4933-8d40-aec9c972988c","Type":"ContainerStarted","Data":"5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f"} Nov 02 20:49:18.294401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.294329 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" event={"ID":"146372b4-37e6-4ca6-976b-5f4cae670a6e","Type":"ContainerStarted","Data":"75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2"} Nov 02 20:49:18.710412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.710325 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:49:18.711557 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.711529 5800 patch_prober.go:29] interesting pod/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 02 20:49:18.711687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.711589 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 02 20:49:18.772381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.772344 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:49:18.776433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.776397 5800 patch_prober.go:29] interesting pod/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 02 20:49:18.776728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:18.776695 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 02 20:49:19.710083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:19.710049 5800 patch_prober.go:29] interesting pod/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 02 20:49:19.710715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:19.710678 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 02 20:49:19.766183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:19.766148 5800 patch_prober.go:29] interesting pod/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 02 20:49:19.766484 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:19.766458 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 02 20:49:20.047308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:20.046437 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:49:20.710973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:20.710936 5800 patch_prober.go:29] interesting pod/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 02 20:49:20.711390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:20.710988 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 02 20:49:20.764227 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:20.764190 5800 patch_prober.go:29] interesting pod/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 02 20:49:20.764548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:20.764519 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 02 20:49:21.058424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:21.058378 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:49:21.710096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:21.710064 5800 patch_prober.go:29] interesting pod/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 02 20:49:21.710312 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:21.710113 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 02 20:49:21.796308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:21.795856 5800 patch_prober.go:29] interesting pod/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 02 20:49:21.796308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:21.795917 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 02 20:49:22.710192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:22.710155 5800 patch_prober.go:29] interesting pod/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 02 20:49:22.710476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:22.710210 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 02 20:49:22.715572 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:49:22.715539 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:49:22.771307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:22.770573 5800 patch_prober.go:29] interesting pod/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 02 20:49:22.771307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:22.770630 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 02 20:49:23.710856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:23.710821 5800 patch_prober.go:29] interesting pod/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 02 20:49:23.711377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:23.710876 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 02 20:49:23.763329 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:23.763265 5800 patch_prober.go:29] interesting pod/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 02 20:49:23.763530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:23.763371 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 02 20:49:24.715384 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:24.715353 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:49:24.716464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:24.716440 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:49:24.768088 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:24.768060 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:49:24.769069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:24.769046 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:49:26.045327 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:26.045293 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd"] Nov 02 20:49:26.092488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:26.092454 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb799601-tdpmd"] Nov 02 20:49:27.626315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:49:27.626268 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd59fbce-d0e5-4ec3-aaad-356fa44ea041" path="/var/lib/kubelet/pods/fd59fbce-d0e5-4ec3-aaad-356fa44ea041/volumes" Nov 02 20:49:52.696268 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:49:52.696232 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:50:19.626267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:19.626184 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9"] Nov 02 20:50:21.089894 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:21.089858 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7"] Nov 02 20:50:21.090391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:21.090067 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" containerID="cri-o://5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f" gracePeriod=30 Nov 02 20:50:22.696982 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:50:22.696942 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:50:24.558595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.558559 5800 scope.go:122] "RemoveContainer" containerID="44098c61bf1af9fb4253ec1d34ce22852392927afd69e876c6788635badd10a3" Nov 02 20:50:24.577668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.577644 5800 scope.go:122] "RemoveContainer" containerID="903fbdb625c03f8c278dd758e23c6f9dc8818b00d031d6fe07c040cc2420e582" Nov 02 20:50:24.626516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.626481 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" podUID="58bbdf74-96a0-4d04-a62a-b6907f04cfca" containerName="payload-processing" containerID="cri-o://2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586" gracePeriod=25 Nov 02 20:50:24.770801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.770764 5800 patch_prober.go:29] interesting pod/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:50:24.770982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.770833 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:50:24.907391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.907362 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:50:24.913658 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.913630 5800 generic.go:356] "Generic (PLEG): container finished" podID="58bbdf74-96a0-4d04-a62a-b6907f04cfca" containerID="2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586" exitCode=0 Nov 02 20:50:24.913810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.913668 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" Nov 02 20:50:24.913810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.913704 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" event={"ID":"58bbdf74-96a0-4d04-a62a-b6907f04cfca","Type":"ContainerDied","Data":"2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586"} Nov 02 20:50:24.913810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.913757 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9" event={"ID":"58bbdf74-96a0-4d04-a62a-b6907f04cfca","Type":"ContainerDied","Data":"facd87d85339b1bab03afdaae0f89af3989c695da7bb990bc844f2bdbdad3735"} Nov 02 20:50:24.913810 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.913780 5800 scope.go:122] "RemoveContainer" containerID="2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586" Nov 02 20:50:24.924896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.924880 5800 scope.go:122] "RemoveContainer" containerID="2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586" Nov 02 20:50:24.925172 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:50:24.925157 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586\": container with ID starting with 2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586 not found: ID does not exist" containerID="2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586" Nov 02 20:50:24.925229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.925180 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586"} err="failed to get container status \"2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586\": rpc error: code = NotFound desc = could not find container \"2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586\": container with ID starting with 2830bfc724985a4bd66202b25e6fbad9ad4fef52b8fbaa117f300e334d357586 not found: ID does not exist" Nov 02 20:50:24.948797 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.948770 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58bbdf74-96a0-4d04-a62a-b6907f04cfca-kube-api-access-hw967\" (UniqueName: \"kubernetes.io/projected/58bbdf74-96a0-4d04-a62a-b6907f04cfca-kube-api-access-hw967\") pod \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " Nov 02 20:50:24.951023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:24.950993 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bbdf74-96a0-4d04-a62a-b6907f04cfca-kube-api-access-hw967" pod "58bbdf74-96a0-4d04-a62a-b6907f04cfca" (UID: "58bbdf74-96a0-4d04-a62a-b6907f04cfca"). InnerVolumeSpecName "kube-api-access-hw967". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:50:25.050143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:25.050100 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58bbdf74-96a0-4d04-a62a-b6907f04cfca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bbdf74-96a0-4d04-a62a-b6907f04cfca-plugins-config-volume\") pod \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\" (UID: \"58bbdf74-96a0-4d04-a62a-b6907f04cfca\") " Nov 02 20:50:25.050616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:25.050589 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bbdf74-96a0-4d04-a62a-b6907f04cfca-plugins-config-volume" pod "58bbdf74-96a0-4d04-a62a-b6907f04cfca" (UID: "58bbdf74-96a0-4d04-a62a-b6907f04cfca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:50:25.050837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:25.050817 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bbdf74-96a0-4d04-a62a-b6907f04cfca-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:25.050910 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:25.050842 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw967\" (UniqueName: \"kubernetes.io/projected/58bbdf74-96a0-4d04-a62a-b6907f04cfca-kube-api-access-hw967\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:25.328967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:25.328933 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9"] Nov 02 20:50:25.336027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:25.335993 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a7c5d1-847967df77-szgl9"] Nov 02 20:50:25.624599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:25.624560 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58bbdf74-96a0-4d04-a62a-b6907f04cfca" path="/var/lib/kubelet/pods/58bbdf74-96a0-4d04-a62a-b6907f04cfca/volumes" Nov 02 20:50:26.370980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.370910 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:50:26.467016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.466985 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c502ded-6a75-4933-8d40-aec9c972988c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c502ded-6a75-4933-8d40-aec9c972988c-istiod-ca-cert\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467023 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-certs\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467083 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-kube-api-access-twn9w\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-kube-api-access-twn9w\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467132 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-istio-token\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-istio-token\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467184 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-data\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467260 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-envoy\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467332 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-socket\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467366 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c502ded-6a75-4933-8d40-aec9c972988c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c502ded-6a75-4933-8d40-aec9c972988c-istio-podinfo\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.467666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467396 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-certs" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:50:26.467666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467410 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c502ded-6a75-4933-8d40-aec9c972988c-istiod-ca-cert" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:50:26.467666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467572 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-data" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:50:26.467666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467581 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-socket" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:50:26.468228 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.467822 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-credential-socket\") pod \"5c502ded-6a75-4933-8d40-aec9c972988c\" (UID: \"5c502ded-6a75-4933-8d40-aec9c972988c\") " Nov 02 20:50:26.468360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.468335 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c502ded-6a75-4933-8d40-aec9c972988c-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.468438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.468397 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.468438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.468412 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.468438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.468426 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.468730 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.468562 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-credential-socket" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:50:26.470391 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.470362 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c502ded-6a75-4933-8d40-aec9c972988c-istio-podinfo" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:50:26.470550 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.470394 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-kube-api-access-twn9w" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "kube-api-access-twn9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:50:26.470652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.470630 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-istio-token" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:50:26.471668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.471642 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-envoy" pod "5c502ded-6a75-4933-8d40-aec9c972988c" (UID: "5c502ded-6a75-4933-8d40-aec9c972988c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:50:26.572859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.572787 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.572859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.572842 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c502ded-6a75-4933-8d40-aec9c972988c-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.572859 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.572857 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c502ded-6a75-4933-8d40-aec9c972988c-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.573154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.572872 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twn9w\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-kube-api-access-twn9w\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.573154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.572904 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c502ded-6a75-4933-8d40-aec9c972988c-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:50:26.932525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.932469 5800 generic.go:356] "Generic (PLEG): container finished" podID="5c502ded-6a75-4933-8d40-aec9c972988c" containerID="5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f" exitCode=0 Nov 02 20:50:26.933006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.932597 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" Nov 02 20:50:26.933006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.932690 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" event={"ID":"5c502ded-6a75-4933-8d40-aec9c972988c","Type":"ContainerDied","Data":"5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f"} Nov 02 20:50:26.933006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.932742 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7" event={"ID":"5c502ded-6a75-4933-8d40-aec9c972988c","Type":"ContainerDied","Data":"32073ffe44e860373f466805bfd2b6feb402a202729a401d499d9864707264a2"} Nov 02 20:50:26.933006 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.932776 5800 scope.go:122] "RemoveContainer" containerID="5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f" Nov 02 20:50:26.954920 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.954895 5800 scope.go:122] "RemoveContainer" containerID="c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f" Nov 02 20:50:26.965887 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.965854 5800 scope.go:122] "RemoveContainer" containerID="5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f" Nov 02 20:50:26.966192 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:50:26.966165 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f\": container with ID starting with 5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f not found: ID does not exist" containerID="5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f" Nov 02 20:50:26.966260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.966206 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f"} err="failed to get container status \"5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f\": rpc error: code = NotFound desc = could not find container \"5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f\": container with ID starting with 5b73a30fea1e9512eeedb57a768616bfa635e1333ad22a9e0c93698cd71a226f not found: ID does not exist" Nov 02 20:50:26.966260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.966233 5800 scope.go:122] "RemoveContainer" containerID="c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f" Nov 02 20:50:26.966562 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:50:26.966527 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f\": container with ID starting with c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f not found: ID does not exist" containerID="c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f" Nov 02 20:50:26.966637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:26.966574 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f"} err="failed to get container status \"c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f\": rpc error: code = NotFound desc = could not find container \"c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f\": container with ID starting with c838c3df68ed42870046c4e7d67fa4d293772327a5c33f86cf52e1650d32a82f not found: ID does not exist" Nov 02 20:50:27.087846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:27.087807 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7"] Nov 02 20:50:27.134582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:27.134551 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-720bda-openshift-default-6c6cd844b7-79xx7"] Nov 02 20:50:27.622818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:27.622784 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" path="/var/lib/kubelet/pods/5c502ded-6a75-4933-8d40-aec9c972988c/volumes" Nov 02 20:50:31.878177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.878139 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t"] Nov 02 20:50:31.878788 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.878773 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" Nov 02 20:50:31.878849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.878789 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" Nov 02 20:50:31.878849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.878828 5800 cpu_manager.go:414] "containerMap: removing container" podUID="58bbdf74-96a0-4d04-a62a-b6907f04cfca" containerName="payload-processing" Nov 02 20:50:31.878849 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.878833 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58bbdf74-96a0-4d04-a62a-b6907f04cfca" containerName="payload-processing" Nov 02 20:50:31.878964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.878939 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" Nov 02 20:50:31.878964 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.878952 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="58bbdf74-96a0-4d04-a62a-b6907f04cfca" containerName="payload-processing" Nov 02 20:50:31.883987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.883965 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:31.935863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.935832 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-28197c\"" Nov 02 20:50:31.936270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.936237 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-28197c-dockercfg-hb5tv\"" Nov 02 20:50:31.942141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.942115 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t"] Nov 02 20:50:31.952896 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.952862 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c323daa6-73b4-45eb-9314-71566a4f2652-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:31.953041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:31.952915 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvhw\" (UniqueName: \"kubernetes.io/projected/c323daa6-73b4-45eb-9314-71566a4f2652-kube-api-access-jtvhw\") pod \"payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:32.058974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.058933 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c323daa6-73b4-45eb-9314-71566a4f2652-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:32.058974 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.058980 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvhw\" (UniqueName: \"kubernetes.io/projected/c323daa6-73b4-45eb-9314-71566a4f2652-kube-api-access-jtvhw\") pod \"payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:32.059967 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.059937 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c323daa6-73b4-45eb-9314-71566a4f2652-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:32.060474 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.060451 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n"] Nov 02 20:50:32.061129 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.061111 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" Nov 02 20:50:32.061217 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.061131 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" Nov 02 20:50:32.061351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.061338 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c502ded-6a75-4933-8d40-aec9c972988c" containerName="istio-proxy" Nov 02 20:50:32.066458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.064937 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.113176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.113141 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvhw\" (UniqueName: \"kubernetes.io/projected/c323daa6-73b4-45eb-9314-71566a4f2652-kube-api-access-jtvhw\") pod \"payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:32.117010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.116957 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n"] Nov 02 20:50:32.160795 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.160723 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5tj\" (UniqueName: \"kubernetes.io/projected/530b676f-60cd-4e6c-8be0-4bc12aa39870-kube-api-access-hp5tj\") pod \"payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.160942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.160852 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530b676f-60cd-4e6c-8be0-4bc12aa39870-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.198371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.198330 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:32.265500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.264128 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5tj\" (UniqueName: \"kubernetes.io/projected/530b676f-60cd-4e6c-8be0-4bc12aa39870-kube-api-access-hp5tj\") pod \"payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.265500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.264366 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530b676f-60cd-4e6c-8be0-4bc12aa39870-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.265500 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.265386 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530b676f-60cd-4e6c-8be0-4bc12aa39870-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.319043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.319009 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5tj\" (UniqueName: \"kubernetes.io/projected/530b676f-60cd-4e6c-8be0-4bc12aa39870-kube-api-access-hp5tj\") pod \"payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.380999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.380957 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:32.570175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.570144 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t"] Nov 02 20:50:32.571882 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:50:32.571855 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc323daa6_73b4_45eb_9314_71566a4f2652.slice/crio-eb618f121d5ae0a4803fe4c95d3e457f6b00c498862aed62d884aab762d722cc WatchSource:0}: Error finding container eb618f121d5ae0a4803fe4c95d3e457f6b00c498862aed62d884aab762d722cc: Status 404 returned error can't find the container with id eb618f121d5ae0a4803fe4c95d3e457f6b00c498862aed62d884aab762d722cc Nov 02 20:50:32.762056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:32.762028 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n"] Nov 02 20:50:32.762408 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:50:32.762382 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530b676f_60cd_4e6c_8be0_4bc12aa39870.slice/crio-c42178a884abbcb14eae198ee0a7b194c246cfa2146e1e020882d7fc9a5302f8 WatchSource:0}: Error finding container c42178a884abbcb14eae198ee0a7b194c246cfa2146e1e020882d7fc9a5302f8: Status 404 returned error can't find the container with id c42178a884abbcb14eae198ee0a7b194c246cfa2146e1e020882d7fc9a5302f8 Nov 02 20:50:33.032636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.032544 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" event={"ID":"c323daa6-73b4-45eb-9314-71566a4f2652","Type":"ContainerStarted","Data":"a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5"} Nov 02 20:50:33.032636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.032625 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:33.033100 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.032650 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" event={"ID":"c323daa6-73b4-45eb-9314-71566a4f2652","Type":"ContainerStarted","Data":"eb618f121d5ae0a4803fe4c95d3e457f6b00c498862aed62d884aab762d722cc"} Nov 02 20:50:33.034303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.034267 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" event={"ID":"530b676f-60cd-4e6c-8be0-4bc12aa39870","Type":"ContainerStarted","Data":"e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e"} Nov 02 20:50:33.034377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.034323 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" event={"ID":"530b676f-60cd-4e6c-8be0-4bc12aa39870","Type":"ContainerStarted","Data":"c42178a884abbcb14eae198ee0a7b194c246cfa2146e1e020882d7fc9a5302f8"} Nov 02 20:50:33.034377 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.034345 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:33.142539 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.142477 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" podStartSLOduration=2.142463175 podStartE2EDuration="2.142463175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:33.138237598 +0000 UTC m=+3731.655010183" watchObservedRunningTime="2026-11-02 20:50:33.142463175 +0000 UTC m=+3731.659235760" Nov 02 20:50:33.245738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:33.245571 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" podStartSLOduration=1.2455508530000001 podStartE2EDuration="1.245550853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:33.242994586 +0000 UTC m=+3731.759767172" watchObservedRunningTime="2026-11-02 20:50:33.245550853 +0000 UTC m=+3731.762323439" Nov 02 20:50:44.046552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:44.046515 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:50:44.047055 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:50:44.046588 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:50:52.697818 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:50:52.697773 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:51:06.044593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:06.044553 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr"] Nov 02 20:51:06.090362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:06.090323 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b381a8f-qc4tr"] Nov 02 20:51:07.043454 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:07.043417 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn"] Nov 02 20:51:07.089307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:07.089246 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e400b3-xjxkn"] Nov 02 20:51:07.624940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:07.624901 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7841753e-6040-44af-b136-73d36c5b9c69" path="/var/lib/kubelet/pods/7841753e-6040-44af-b136-73d36c5b9c69/volumes" Nov 02 20:51:07.625510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:07.625486 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4cf8c34-b36b-4cd3-8105-c41f93f7a672" path="/var/lib/kubelet/pods/d4cf8c34-b36b-4cd3-8105-c41f93f7a672/volumes" Nov 02 20:51:09.044031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:09.043992 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt"] Nov 02 20:51:09.090593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:09.090552 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a202f3e-5ktbt"] Nov 02 20:51:09.622447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:09.622406 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b770812-57da-48ea-b100-9fc54a7c181f" path="/var/lib/kubelet/pods/2b770812-57da-48ea-b100-9fc54a7c181f/volumes" Nov 02 20:51:17.044808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:17.044770 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4"] Nov 02 20:51:17.092696 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:17.092662 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12234760-k75c4"] Nov 02 20:51:17.624875 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:17.624840 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c57e2f-6f21-4a3c-87d2-2df57937c4c7" path="/var/lib/kubelet/pods/c1c57e2f-6f21-4a3c-87d2-2df57937c4c7/volumes" Nov 02 20:51:18.048291 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:18.048242 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68"] Nov 02 20:51:18.091461 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:18.091432 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e265790a-fzv68"] Nov 02 20:51:19.623136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:19.623102 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06636ce0-2746-4dfd-9cff-09252b1b3a18" path="/var/lib/kubelet/pods/06636ce0-2746-4dfd-9cff-09252b1b3a18/volumes" Nov 02 20:51:20.043362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:20.043323 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc"] Nov 02 20:51:20.089173 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:20.089137 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1aa9ee61-6lbkc"] Nov 02 20:51:21.623495 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:21.623462 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000ccce9-467a-4c57-a38c-f5116dedf5f8" path="/var/lib/kubelet/pods/000ccce9-467a-4c57-a38c-f5116dedf5f8/volumes" Nov 02 20:51:22.699991 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:51:22.699951 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:51:31.534097 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:31.534010 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9"] Nov 02 20:51:31.633408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:31.633371 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl"] Nov 02 20:51:36.534992 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.534950 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" podUID="6a016fb4-e940-4d5c-a282-b541fc3957c2" containerName="payload-processing" containerID="cri-o://5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf" gracePeriod=25 Nov 02 20:51:36.633759 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.633721 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" podUID="393642a7-a1cb-4e36-abda-e77651962708" containerName="payload-pre-processing" containerID="cri-o://c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3" gracePeriod=25 Nov 02 20:51:36.813448 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.813416 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:51:36.925074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.924914 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a016fb4-e940-4d5c-a282-b541fc3957c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a016fb4-e940-4d5c-a282-b541fc3957c2-plugins-config-volume\") pod \"6a016fb4-e940-4d5c-a282-b541fc3957c2\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " Nov 02 20:51:36.925074 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.925021 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a016fb4-e940-4d5c-a282-b541fc3957c2-kube-api-access-22rbr\" (UniqueName: \"kubernetes.io/projected/6a016fb4-e940-4d5c-a282-b541fc3957c2-kube-api-access-22rbr\") pod \"6a016fb4-e940-4d5c-a282-b541fc3957c2\" (UID: \"6a016fb4-e940-4d5c-a282-b541fc3957c2\") " Nov 02 20:51:36.925442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.925411 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a016fb4-e940-4d5c-a282-b541fc3957c2-plugins-config-volume" pod "6a016fb4-e940-4d5c-a282-b541fc3957c2" (UID: "6a016fb4-e940-4d5c-a282-b541fc3957c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:51:36.925548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.925534 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a016fb4-e940-4d5c-a282-b541fc3957c2-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:36.927638 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.927612 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a016fb4-e940-4d5c-a282-b541fc3957c2-kube-api-access-22rbr" pod "6a016fb4-e940-4d5c-a282-b541fc3957c2" (UID: "6a016fb4-e940-4d5c-a282-b541fc3957c2"). InnerVolumeSpecName "kube-api-access-22rbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:51:36.937194 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:36.937166 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:51:37.026717 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.026689 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393642a7-a1cb-4e36-abda-e77651962708-kube-api-access-mkpg9\" (UniqueName: \"kubernetes.io/projected/393642a7-a1cb-4e36-abda-e77651962708-kube-api-access-mkpg9\") pod \"393642a7-a1cb-4e36-abda-e77651962708\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " Nov 02 20:51:37.026925 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.026835 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393642a7-a1cb-4e36-abda-e77651962708-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393642a7-a1cb-4e36-abda-e77651962708-plugins-config-volume\") pod \"393642a7-a1cb-4e36-abda-e77651962708\" (UID: \"393642a7-a1cb-4e36-abda-e77651962708\") " Nov 02 20:51:37.027141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.027126 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22rbr\" (UniqueName: \"kubernetes.io/projected/6a016fb4-e940-4d5c-a282-b541fc3957c2-kube-api-access-22rbr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:37.027219 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.027166 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393642a7-a1cb-4e36-abda-e77651962708-plugins-config-volume" pod "393642a7-a1cb-4e36-abda-e77651962708" (UID: "393642a7-a1cb-4e36-abda-e77651962708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:51:37.028869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.028845 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393642a7-a1cb-4e36-abda-e77651962708-kube-api-access-mkpg9" pod "393642a7-a1cb-4e36-abda-e77651962708" (UID: "393642a7-a1cb-4e36-abda-e77651962708"). InnerVolumeSpecName "kube-api-access-mkpg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:51:37.128673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.128595 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkpg9\" (UniqueName: \"kubernetes.io/projected/393642a7-a1cb-4e36-abda-e77651962708-kube-api-access-mkpg9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:37.128673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.128622 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393642a7-a1cb-4e36-abda-e77651962708-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:37.601020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.600986 5800 generic.go:356] "Generic (PLEG): container finished" podID="6a016fb4-e940-4d5c-a282-b541fc3957c2" containerID="5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf" exitCode=0 Nov 02 20:51:37.601504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.601032 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" Nov 02 20:51:37.601504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.601071 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" event={"ID":"6a016fb4-e940-4d5c-a282-b541fc3957c2","Type":"ContainerDied","Data":"5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf"} Nov 02 20:51:37.601504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.601123 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9" event={"ID":"6a016fb4-e940-4d5c-a282-b541fc3957c2","Type":"ContainerDied","Data":"f231df43a8c003fa53e32a6dcb7763589a3d09bdfa83c58d504f17c20699afd9"} Nov 02 20:51:37.601504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.601147 5800 scope.go:122] "RemoveContainer" containerID="5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf" Nov 02 20:51:37.602879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.602830 5800 generic.go:356] "Generic (PLEG): container finished" podID="393642a7-a1cb-4e36-abda-e77651962708" containerID="c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3" exitCode=0 Nov 02 20:51:37.602879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.602861 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" event={"ID":"393642a7-a1cb-4e36-abda-e77651962708","Type":"ContainerDied","Data":"c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3"} Nov 02 20:51:37.603013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.602886 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" Nov 02 20:51:37.603013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.602898 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl" event={"ID":"393642a7-a1cb-4e36-abda-e77651962708","Type":"ContainerDied","Data":"748bfa2a831a9c722568886cf9c7b483c1a302fd0969c9533d58a7439fcfbd5f"} Nov 02 20:51:37.614598 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.614572 5800 scope.go:122] "RemoveContainer" containerID="5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf" Nov 02 20:51:37.618979 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:51:37.618943 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf\": container with ID starting with 5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf not found: ID does not exist" containerID="5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf" Nov 02 20:51:37.619086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.618989 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf"} err="failed to get container status \"5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf\": rpc error: code = NotFound desc = could not find container \"5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf\": container with ID starting with 5f254238971715259cee40848c70bd01026dfa2ca6f352a65447e459409b32bf not found: ID does not exist" Nov 02 20:51:37.619086 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.619017 5800 scope.go:122] "RemoveContainer" containerID="c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3" Nov 02 20:51:37.641401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.641368 5800 scope.go:122] "RemoveContainer" containerID="c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3" Nov 02 20:51:37.641665 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:51:37.641645 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3\": container with ID starting with c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3 not found: ID does not exist" containerID="c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3" Nov 02 20:51:37.641742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.641677 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3"} err="failed to get container status \"c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3\": rpc error: code = NotFound desc = could not find container \"c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3\": container with ID starting with c29312c8161582a8bf955b21513e969cef8fd42bd09f29f71d5fc20678d420e3 not found: ID does not exist" Nov 02 20:51:37.755677 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.755643 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl"] Nov 02 20:51:37.801816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.801782 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0a8f62-7b6c579995-r9kfl"] Nov 02 20:51:37.859411 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.859322 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9"] Nov 02 20:51:37.862162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:37.862139 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0a8f62-55f87c765c-68nr9"] Nov 02 20:51:38.343509 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:38.343476 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v"] Nov 02 20:51:39.624668 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:39.624638 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393642a7-a1cb-4e36-abda-e77651962708" path="/var/lib/kubelet/pods/393642a7-a1cb-4e36-abda-e77651962708/volumes" Nov 02 20:51:39.625077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:39.625063 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a016fb4-e940-4d5c-a282-b541fc3957c2" path="/var/lib/kubelet/pods/6a016fb4-e940-4d5c-a282-b541fc3957c2/volumes" Nov 02 20:51:41.457962 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.457926 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s"] Nov 02 20:51:41.459578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.459550 5800 cpu_manager.go:414] "containerMap: removing container" podUID="6a016fb4-e940-4d5c-a282-b541fc3957c2" containerName="payload-processing" Nov 02 20:51:41.459754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.459740 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a016fb4-e940-4d5c-a282-b541fc3957c2" containerName="payload-processing" Nov 02 20:51:41.459908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.459896 5800 cpu_manager.go:414] "containerMap: removing container" podUID="393642a7-a1cb-4e36-abda-e77651962708" containerName="payload-pre-processing" Nov 02 20:51:41.459989 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.459978 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393642a7-a1cb-4e36-abda-e77651962708" containerName="payload-pre-processing" Nov 02 20:51:41.460359 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.460340 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="393642a7-a1cb-4e36-abda-e77651962708" containerName="payload-pre-processing" Nov 02 20:51:41.460459 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.460396 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a016fb4-e940-4d5c-a282-b541fc3957c2" containerName="payload-processing" Nov 02 20:51:41.465622 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.465600 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.514504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.514465 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s"] Nov 02 20:51:41.556382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.556343 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:51:41.556382 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.556370 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:51:41.556608 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.556399 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:51:41.560411 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.560395 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:51:41.594157 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.594126 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcgq\" (UniqueName: \"kubernetes.io/projected/c6dcb9ea-bfec-441e-84c0-5386de3141b4-kube-api-access-jvcgq\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.594365 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.594338 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.594442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.594394 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.695829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.695795 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.696008 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.695916 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcgq\" (UniqueName: \"kubernetes.io/projected/c6dcb9ea-bfec-441e-84c0-5386de3141b4-kube-api-access-jvcgq\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.696062 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.696049 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.696726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.696699 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.698199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.698181 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.748918 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.748858 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcgq\" (UniqueName: \"kubernetes.io/projected/c6dcb9ea-bfec-441e-84c0-5386de3141b4-kube-api-access-jvcgq\") pod \"maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:41.779595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:41.779569 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:42.147460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:42.147429 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s"] Nov 02 20:51:42.149354 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:51:42.149321 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6dcb9ea_bfec_441e_84c0_5386de3141b4.slice/crio-b745226c11731591a6b9e9270f2d255f0263c80246aea5cc33c3cd51569019a3 WatchSource:0}: Error finding container b745226c11731591a6b9e9270f2d255f0263c80246aea5cc33c3cd51569019a3: Status 404 returned error can't find the container with id b745226c11731591a6b9e9270f2d255f0263c80246aea5cc33c3cd51569019a3 Nov 02 20:51:42.641458 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:42.641420 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" event={"ID":"c6dcb9ea-bfec-441e-84c0-5386de3141b4","Type":"ContainerStarted","Data":"eb61a26bdafdcb56f4bffbc21aa94bba520290482708893ab2328df6e622eff8"} Nov 02 20:51:42.641871 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:42.641479 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" event={"ID":"c6dcb9ea-bfec-441e-84c0-5386de3141b4","Type":"ContainerStarted","Data":"b745226c11731591a6b9e9270f2d255f0263c80246aea5cc33c3cd51569019a3"} Nov 02 20:51:42.749771 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:42.749725 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" podStartSLOduration=1.7497112609999999 podStartE2EDuration="1.749711261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:51:42.74744421 +0000 UTC m=+3801.264216795" watchObservedRunningTime="2026-11-02 20:51:42.749711261 +0000 UTC m=+3801.266483845" Nov 02 20:51:43.356185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.344178 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" podUID="9169b8b7-3395-4ec5-92b8-dcdef846353f" containerName="payload-processing" containerID="cri-o://eb19c6649098a2141e6a5d77fe1b0443dc5a2fbf736e45397877e06aec65131a" gracePeriod=25 Nov 02 20:51:43.650408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.650377 5800 generic.go:356] "Generic (PLEG): container finished" podID="9169b8b7-3395-4ec5-92b8-dcdef846353f" containerID="eb19c6649098a2141e6a5d77fe1b0443dc5a2fbf736e45397877e06aec65131a" exitCode=0 Nov 02 20:51:43.650839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.650465 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" event={"ID":"9169b8b7-3395-4ec5-92b8-dcdef846353f","Type":"ContainerDied","Data":"eb19c6649098a2141e6a5d77fe1b0443dc5a2fbf736e45397877e06aec65131a"} Nov 02 20:51:43.650839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.650519 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" event={"ID":"9169b8b7-3395-4ec5-92b8-dcdef846353f","Type":"ContainerDied","Data":"d74924beba77022d3c816eedf0049ef823cc34f9002bf69b1892b04356b734dd"} Nov 02 20:51:43.650839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.650546 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d74924beba77022d3c816eedf0049ef823cc34f9002bf69b1892b04356b734dd" Nov 02 20:51:43.652330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.652270 5800 generic.go:356] "Generic (PLEG): container finished" podID="c6dcb9ea-bfec-441e-84c0-5386de3141b4" containerID="eb61a26bdafdcb56f4bffbc21aa94bba520290482708893ab2328df6e622eff8" exitCode=0 Nov 02 20:51:43.652442 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.652328 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" event={"ID":"c6dcb9ea-bfec-441e-84c0-5386de3141b4","Type":"ContainerDied","Data":"eb61a26bdafdcb56f4bffbc21aa94bba520290482708893ab2328df6e622eff8"} Nov 02 20:51:43.657987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.657965 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:51:43.694226 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.694194 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn"] Nov 02 20:51:43.694864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.694837 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9169b8b7-3395-4ec5-92b8-dcdef846353f" containerName="payload-processing" Nov 02 20:51:43.694864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.694851 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9169b8b7-3395-4ec5-92b8-dcdef846353f" containerName="payload-processing" Nov 02 20:51:43.694961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.694954 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9169b8b7-3395-4ec5-92b8-dcdef846353f" containerName="payload-processing" Nov 02 20:51:43.698408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.698391 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.722911 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.722888 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9169b8b7-3395-4ec5-92b8-dcdef846353f-kube-api-access-6s9tz\" (UniqueName: \"kubernetes.io/projected/9169b8b7-3395-4ec5-92b8-dcdef846353f-kube-api-access-6s9tz\") pod \"9169b8b7-3395-4ec5-92b8-dcdef846353f\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " Nov 02 20:51:43.723050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.722928 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9169b8b7-3395-4ec5-92b8-dcdef846353f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9169b8b7-3395-4ec5-92b8-dcdef846353f-plugins-config-volume\") pod \"9169b8b7-3395-4ec5-92b8-dcdef846353f\" (UID: \"9169b8b7-3395-4ec5-92b8-dcdef846353f\") " Nov 02 20:51:43.723362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.723341 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9169b8b7-3395-4ec5-92b8-dcdef846353f-plugins-config-volume" pod "9169b8b7-3395-4ec5-92b8-dcdef846353f" (UID: "9169b8b7-3395-4ec5-92b8-dcdef846353f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:51:43.725042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.725019 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9169b8b7-3395-4ec5-92b8-dcdef846353f-kube-api-access-6s9tz" pod "9169b8b7-3395-4ec5-92b8-dcdef846353f" (UID: "9169b8b7-3395-4ec5-92b8-dcdef846353f"). InnerVolumeSpecName "kube-api-access-6s9tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:51:43.750387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.750361 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn"] Nov 02 20:51:43.824187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.824146 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8flr\" (UniqueName: \"kubernetes.io/projected/61215f70-b69f-4da0-85e3-a9616a6d638b-kube-api-access-l8flr\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.824418 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.824309 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.824498 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.824422 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.824676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.824647 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s9tz\" (UniqueName: \"kubernetes.io/projected/9169b8b7-3395-4ec5-92b8-dcdef846353f-kube-api-access-6s9tz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:43.824798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.824677 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9169b8b7-3395-4ec5-92b8-dcdef846353f-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:43.925883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.925789 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8flr\" (UniqueName: \"kubernetes.io/projected/61215f70-b69f-4da0-85e3-a9616a6d638b-kube-api-access-l8flr\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.926069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.925894 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.926069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.925943 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.926735 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.926713 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.928400 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.928379 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:43.978605 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:43.978570 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8flr\" (UniqueName: \"kubernetes.io/projected/61215f70-b69f-4da0-85e3-a9616a6d638b-kube-api-access-l8flr\") pod \"maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:44.013060 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.013027 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:44.385180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.385153 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn"] Nov 02 20:51:44.386293 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:51:44.386254 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61215f70_b69f_4da0_85e3_a9616a6d638b.slice/crio-9bb2bb5d017ad83bc8e690157aee64e392eaf930e70f49f90323889ce961f105 WatchSource:0}: Error finding container 9bb2bb5d017ad83bc8e690157aee64e392eaf930e70f49f90323889ce961f105: Status 404 returned error can't find the container with id 9bb2bb5d017ad83bc8e690157aee64e392eaf930e70f49f90323889ce961f105 Nov 02 20:51:44.660889 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.660858 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" event={"ID":"61215f70-b69f-4da0-85e3-a9616a6d638b","Type":"ContainerStarted","Data":"51b87207758e2a3ab575a9674421c61b5bbbdd73dad88b7dbd409450b120abef"} Nov 02 20:51:44.661341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.660917 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" event={"ID":"61215f70-b69f-4da0-85e3-a9616a6d638b","Type":"ContainerStarted","Data":"9bb2bb5d017ad83bc8e690157aee64e392eaf930e70f49f90323889ce961f105"} Nov 02 20:51:44.661341 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.661117 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v" Nov 02 20:51:44.770908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.770842 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" podStartSLOduration=1.7708223969999999 podStartE2EDuration="1.770822397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:51:44.769452598 +0000 UTC m=+3803.286225183" watchObservedRunningTime="2026-11-02 20:51:44.770822397 +0000 UTC m=+3803.287594983" Nov 02 20:51:44.819684 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.819658 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:44.879234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.879201 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v"] Nov 02 20:51:44.882212 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.882191 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8d5148-694b7b999f-zlg9v"] Nov 02 20:51:44.938072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.937993 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-openshift-ca\") pod \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " Nov 02 20:51:44.938072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.938061 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-xks-ca\") pod \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " Nov 02 20:51:44.938272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.938261 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6dcb9ea-bfec-441e-84c0-5386de3141b4-kube-api-access-jvcgq\" (UniqueName: \"kubernetes.io/projected/c6dcb9ea-bfec-441e-84c0-5386de3141b4-kube-api-access-jvcgq\") pod \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\" (UID: \"c6dcb9ea-bfec-441e-84c0-5386de3141b4\") " Nov 02 20:51:44.938364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.938342 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-openshift-ca" pod "c6dcb9ea-bfec-441e-84c0-5386de3141b4" (UID: "c6dcb9ea-bfec-441e-84c0-5386de3141b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:51:44.938743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.938719 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:44.940103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.940080 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-xks-ca" pod "c6dcb9ea-bfec-441e-84c0-5386de3141b4" (UID: "c6dcb9ea-bfec-441e-84c0-5386de3141b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:51:44.940261 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:44.940243 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dcb9ea-bfec-441e-84c0-5386de3141b4-kube-api-access-jvcgq" pod "c6dcb9ea-bfec-441e-84c0-5386de3141b4" (UID: "c6dcb9ea-bfec-441e-84c0-5386de3141b4"). InnerVolumeSpecName "kube-api-access-jvcgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:51:45.040493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.040457 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6dcb9ea-bfec-441e-84c0-5386de3141b4-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:45.040493 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.040487 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcgq\" (UniqueName: \"kubernetes.io/projected/c6dcb9ea-bfec-441e-84c0-5386de3141b4-kube-api-access-jvcgq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:45.624056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.623978 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9169b8b7-3395-4ec5-92b8-dcdef846353f" path="/var/lib/kubelet/pods/9169b8b7-3395-4ec5-92b8-dcdef846353f/volumes" Nov 02 20:51:45.672845 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.672812 5800 generic.go:356] "Generic (PLEG): container finished" podID="61215f70-b69f-4da0-85e3-a9616a6d638b" containerID="51b87207758e2a3ab575a9674421c61b5bbbdd73dad88b7dbd409450b120abef" exitCode=0 Nov 02 20:51:45.673253 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.672899 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" event={"ID":"61215f70-b69f-4da0-85e3-a9616a6d638b","Type":"ContainerDied","Data":"51b87207758e2a3ab575a9674421c61b5bbbdd73dad88b7dbd409450b120abef"} Nov 02 20:51:45.674398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.674380 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" Nov 02 20:51:45.674398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.674388 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s" event={"ID":"c6dcb9ea-bfec-441e-84c0-5386de3141b4","Type":"ContainerDied","Data":"b745226c11731591a6b9e9270f2d255f0263c80246aea5cc33c3cd51569019a3"} Nov 02 20:51:45.674536 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:45.674422 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b745226c11731591a6b9e9270f2d255f0263c80246aea5cc33c3cd51569019a3" Nov 02 20:51:46.845831 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.845808 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:46.963783 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.963698 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-xks-ca\") pod \"61215f70-b69f-4da0-85e3-a9616a6d638b\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " Nov 02 20:51:46.963948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.963886 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-openshift-ca\") pod \"61215f70-b69f-4da0-85e3-a9616a6d638b\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " Nov 02 20:51:46.963948 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.963932 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61215f70-b69f-4da0-85e3-a9616a6d638b-kube-api-access-l8flr\" (UniqueName: \"kubernetes.io/projected/61215f70-b69f-4da0-85e3-a9616a6d638b-kube-api-access-l8flr\") pod \"61215f70-b69f-4da0-85e3-a9616a6d638b\" (UID: \"61215f70-b69f-4da0-85e3-a9616a6d638b\") " Nov 02 20:51:46.964372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.964346 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-openshift-ca" pod "61215f70-b69f-4da0-85e3-a9616a6d638b" (UID: "61215f70-b69f-4da0-85e3-a9616a6d638b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:51:46.964515 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.964494 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:46.966490 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.966461 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-xks-ca" pod "61215f70-b69f-4da0-85e3-a9616a6d638b" (UID: "61215f70-b69f-4da0-85e3-a9616a6d638b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:51:46.966812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:46.966792 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61215f70-b69f-4da0-85e3-a9616a6d638b-kube-api-access-l8flr" pod "61215f70-b69f-4da0-85e3-a9616a6d638b" (UID: "61215f70-b69f-4da0-85e3-a9616a6d638b"). InnerVolumeSpecName "kube-api-access-l8flr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:51:47.065876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:47.065841 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61215f70-b69f-4da0-85e3-a9616a6d638b-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:47.065876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:47.065872 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8flr\" (UniqueName: \"kubernetes.io/projected/61215f70-b69f-4da0-85e3-a9616a6d638b-kube-api-access-l8flr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:51:47.690980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:47.690950 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" Nov 02 20:51:47.690980 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:47.690970 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn" event={"ID":"61215f70-b69f-4da0-85e3-a9616a6d638b","Type":"ContainerDied","Data":"9bb2bb5d017ad83bc8e690157aee64e392eaf930e70f49f90323889ce961f105"} Nov 02 20:51:47.691199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:47.691018 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb2bb5d017ad83bc8e690157aee64e392eaf930e70f49f90323889ce961f105" Nov 02 20:51:52.696707 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:51:52.696669 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:51:55.301467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:55.301428 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s"] Nov 02 20:51:55.346649 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:55.346614 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a7c5d1-jr66s"] Nov 02 20:51:55.704714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:55.701443 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6dcb9ea-bfec-441e-84c0-5386de3141b4" path="/var/lib/kubelet/pods/c6dcb9ea-bfec-441e-84c0-5386de3141b4/volumes" Nov 02 20:51:56.623423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:56.623389 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn"] Nov 02 20:51:56.669455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:56.669423 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-608c11-qvmdn"] Nov 02 20:51:56.797955 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:56.797914 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp"] Nov 02 20:51:56.798199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:56.798167 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" containerID="cri-o://75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2" gracePeriod=30 Nov 02 20:51:57.624414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:51:57.624381 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61215f70-b69f-4da0-85e3-a9616a6d638b" path="/var/lib/kubelet/pods/61215f70-b69f-4da0-85e3-a9616a6d638b/volumes" Nov 02 20:52:01.881688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.881653 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75574cc589-7q5dn"] Nov 02 20:52:01.882375 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.882360 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c6dcb9ea-bfec-441e-84c0-5386de3141b4" containerName="revoke-keys" Nov 02 20:52:01.882428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.882376 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6dcb9ea-bfec-441e-84c0-5386de3141b4" containerName="revoke-keys" Nov 02 20:52:01.882428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.882386 5800 cpu_manager.go:414] "containerMap: removing container" podUID="61215f70-b69f-4da0-85e3-a9616a6d638b" containerName="revoke-keys" Nov 02 20:52:01.882428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.882391 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61215f70-b69f-4da0-85e3-a9616a6d638b" containerName="revoke-keys" Nov 02 20:52:01.882538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.882517 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6dcb9ea-bfec-441e-84c0-5386de3141b4" containerName="revoke-keys" Nov 02 20:52:01.882538 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.882524 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="61215f70-b69f-4da0-85e3-a9616a6d638b" containerName="revoke-keys" Nov 02 20:52:01.885959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.885943 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:01.939487 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.939435 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2da8cddd53dd\"" Nov 02 20:52:01.960403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.958488 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75574cc589-7q5dn"] Nov 02 20:52:01.971874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.971838 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-provider-credentials\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:01.972163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.972142 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-routing-overlay-volume\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:01.972340 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.972323 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2g9\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-kube-api-access-cw2g9\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:01.972467 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.972453 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-plugins-config-volume\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:01.980666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.980635 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lkhrv\"" Nov 02 20:52:01.980812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.980757 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 20:52:01.980812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.980642 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 20:52:01.980937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.980911 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 20:52:01.981080 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:01.981062 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 20:52:02.073794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.073694 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-routing-overlay-volume\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.073794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.073761 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2g9\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-kube-api-access-cw2g9\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.073794 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.073786 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-plugins-config-volume\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.074106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.073987 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-provider-credentials\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.075537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.075501 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-plugins-config-volume\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.075699 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.075520 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-routing-overlay-volume\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.076787 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.076759 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-provider-credentials\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.147822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.147787 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2g9\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-kube-api-access-cw2g9\") pod \"payload-processing-external-model-75574cc589-7q5dn\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.191832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.191802 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:52:02.206565 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.206533 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.277712 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-kube-api-access-gk5tx\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-kube-api-access-gk5tx\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.277836 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-certs\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.277874 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-socket\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.277946 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-credential-socket\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.277985 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-envoy\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.278055 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-data\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.278100 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-token\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-token\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.278187 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-podinfo\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.279001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.278237 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146372b4-37e6-4ca6-976b-5f4cae670a6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146372b4-37e6-4ca6-976b-5f4cae670a6e-istiod-ca-cert\") pod \"146372b4-37e6-4ca6-976b-5f4cae670a6e\" (UID: \"146372b4-37e6-4ca6-976b-5f4cae670a6e\") " Nov 02 20:52:02.280364 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.279964 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-credential-socket" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:02.281394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.280744 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-certs" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:02.281394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.281109 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-socket" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:02.281954 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.281652 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-data" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:02.292318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.282253 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146372b4-37e6-4ca6-976b-5f4cae670a6e-istiod-ca-cert" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:52:02.292318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.283000 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-kube-api-access-gk5tx" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "kube-api-access-gk5tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:02.292318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.285257 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-token" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:02.292318 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.287333 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-envoy" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:02.298348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.295162 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-podinfo" pod "146372b4-37e6-4ca6-976b-5f4cae670a6e" (UID: "146372b4-37e6-4ca6-976b-5f4cae670a6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:52:02.380543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380512 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380541 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380550 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380559 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146372b4-37e6-4ca6-976b-5f4cae670a6e-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380569 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146372b4-37e6-4ca6-976b-5f4cae670a6e-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380580 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5tx\" (UniqueName: \"kubernetes.io/projected/146372b4-37e6-4ca6-976b-5f4cae670a6e-kube-api-access-gk5tx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380590 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380598 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.380774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.380607 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146372b4-37e6-4ca6-976b-5f4cae670a6e-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:02.663560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.663532 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75574cc589-7q5dn"] Nov 02 20:52:02.664778 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:52:02.664752 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fedef7_fe57_45be_b0db_cd48a2905442.slice/crio-4c636f9e03f760b470fb3803da2a87d465d202bf648ebbcc4eabcdb9736855ed WatchSource:0}: Error finding container 4c636f9e03f760b470fb3803da2a87d465d202bf648ebbcc4eabcdb9736855ed: Status 404 returned error can't find the container with id 4c636f9e03f760b470fb3803da2a87d465d202bf648ebbcc4eabcdb9736855ed Nov 02 20:52:02.820687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.820650 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" event={"ID":"17fedef7-fe57-45be-b0db-cd48a2905442","Type":"ContainerStarted","Data":"44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6"} Nov 02 20:52:02.820862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.820722 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:02.820862 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.820761 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" event={"ID":"17fedef7-fe57-45be-b0db-cd48a2905442","Type":"ContainerStarted","Data":"4c636f9e03f760b470fb3803da2a87d465d202bf648ebbcc4eabcdb9736855ed"} Nov 02 20:52:02.822738 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.822707 5800 generic.go:356] "Generic (PLEG): container finished" podID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerID="75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2" exitCode=0 Nov 02 20:52:02.822886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.822756 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" Nov 02 20:52:02.822886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.822784 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" event={"ID":"146372b4-37e6-4ca6-976b-5f4cae670a6e","Type":"ContainerDied","Data":"75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2"} Nov 02 20:52:02.822886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.822847 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp" event={"ID":"146372b4-37e6-4ca6-976b-5f4cae670a6e","Type":"ContainerDied","Data":"0b250af6937173103233a883465b30b99dc950b797bc81d93e8370c233b96d5e"} Nov 02 20:52:02.822886 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.822880 5800 scope.go:122] "RemoveContainer" containerID="75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2" Nov 02 20:52:02.839563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.839544 5800 scope.go:122] "RemoveContainer" containerID="75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e" Nov 02 20:52:02.849200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.849180 5800 scope.go:122] "RemoveContainer" containerID="75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2" Nov 02 20:52:02.849530 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:52:02.849509 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2\": container with ID starting with 75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2 not found: ID does not exist" containerID="75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2" Nov 02 20:52:02.849618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.849538 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2"} err="failed to get container status \"75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2\": rpc error: code = NotFound desc = could not find container \"75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2\": container with ID starting with 75607cadded71d7bad97628a6cf5d2f5d436221754e9501a3578e976732037c2 not found: ID does not exist" Nov 02 20:52:02.849618 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.849557 5800 scope.go:122] "RemoveContainer" containerID="75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e" Nov 02 20:52:02.849810 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:52:02.849788 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e\": container with ID starting with 75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e not found: ID does not exist" containerID="75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e" Nov 02 20:52:02.849869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.849821 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e"} err="failed to get container status \"75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e\": rpc error: code = NotFound desc = could not find container \"75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e\": container with ID starting with 75d2110b45fb3fb7a3a7e899e15c9b5f52072b186163c2246b7703752a4b4d5e not found: ID does not exist" Nov 02 20:52:02.929505 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:02.929406 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" podStartSLOduration=1.929392044 podStartE2EDuration="1.929392044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:02.926006228 +0000 UTC m=+3821.442778813" watchObservedRunningTime="2026-11-02 20:52:02.929392044 +0000 UTC m=+3821.446164665" Nov 02 20:52:03.032058 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:03.032023 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp"] Nov 02 20:52:03.078376 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:03.078343 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-608c11-openshift-default-7768f9df5c-bx7gp"] Nov 02 20:52:03.625160 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:03.625124 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" path="/var/lib/kubelet/pods/146372b4-37e6-4ca6-976b-5f4cae670a6e/volumes" Nov 02 20:52:10.208503 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.208467 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c"] Nov 02 20:52:10.209123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.209108 5800 cpu_manager.go:414] "containerMap: removing container" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" Nov 02 20:52:10.209170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.209124 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" Nov 02 20:52:10.209170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.209147 5800 cpu_manager.go:414] "containerMap: removing container" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" Nov 02 20:52:10.209170 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.209153 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" Nov 02 20:52:10.209272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.209252 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" Nov 02 20:52:10.214338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.214318 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.268431 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.268392 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c"] Nov 02 20:52:10.283139 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.283109 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-routing-overlay-volume\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.283322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.283148 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-plugins-config-volume\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.283322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.283170 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dntvh\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-kube-api-access-dntvh\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.283580 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.283549 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-provider-credentials\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.309075 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.309041 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c233618dc3e5\"" Nov 02 20:52:10.384985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.384946 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-provider-credentials\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.384985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.384991 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-routing-overlay-volume\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.385270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.385016 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-plugins-config-volume\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.385270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.385055 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dntvh\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-kube-api-access-dntvh\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.385669 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.385645 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-routing-overlay-volume\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.385781 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.385760 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-plugins-config-volume\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.387585 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.387564 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-provider-credentials\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.439143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.439095 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dntvh\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-kube-api-access-dntvh\") pod \"payload-processing-external-model-544bc8ddf6-fdv4c\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.528026 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.527996 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:10.906072 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:10.906029 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c"] Nov 02 20:52:10.906619 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:52:10.906591 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739dc8e6_1fac_45bb_b38f_38b745c0f627.slice/crio-78dd3b9187321b3cee397b8638d2c7823e16cf618a777df83c90d26fd6faf0df WatchSource:0}: Error finding container 78dd3b9187321b3cee397b8638d2c7823e16cf618a777df83c90d26fd6faf0df: Status 404 returned error can't find the container with id 78dd3b9187321b3cee397b8638d2c7823e16cf618a777df83c90d26fd6faf0df Nov 02 20:52:11.900959 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:11.900918 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" event={"ID":"739dc8e6-1fac-45bb-b38f-38b745c0f627","Type":"ContainerStarted","Data":"2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83"} Nov 02 20:52:11.901526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:11.900983 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" event={"ID":"739dc8e6-1fac-45bb-b38f-38b745c0f627","Type":"ContainerStarted","Data":"78dd3b9187321b3cee397b8638d2c7823e16cf618a777df83c90d26fd6faf0df"} Nov 02 20:52:11.901526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:11.901133 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:12.030990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:12.028985 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" podStartSLOduration=2.028963709 podStartE2EDuration="2.028963709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:12.028358145 +0000 UTC m=+3830.545130731" watchObservedRunningTime="2026-11-02 20:52:12.028963709 +0000 UTC m=+3830.545736295" Nov 02 20:52:13.836044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:13.836014 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:22.697218 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:52:22.697182 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:52:22.914907 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:22.914869 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:52:23.071371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:23.071339 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75574cc589-7q5dn"] Nov 02 20:52:24.645705 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:24.645670 5800 scope.go:122] "RemoveContainer" containerID="ea664614dcd5aa02eb11578b07ec5303ed381d01a1a15c8ff4494da59287d203" Nov 02 20:52:24.658260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:24.658238 5800 scope.go:122] "RemoveContainer" containerID="383abece7e9980e5a01f9fbabc0a916e9f38d0717940c59a46c4143b791ae408" Nov 02 20:52:24.668864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:24.668845 5800 scope.go:122] "RemoveContainer" containerID="8072bbb1850685bc143da23bfbafcd26f010ee7def31f0069dc5f87b013da13c" Nov 02 20:52:24.678590 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:24.678557 5800 scope.go:122] "RemoveContainer" containerID="42bf7038f11ecfe3197786a952a8ae2b06aad5b4e9c4259be2e833ef66ad60f2" Nov 02 20:52:24.688168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:24.688147 5800 scope.go:122] "RemoveContainer" containerID="5d215b9386af05a8f7b4d7f32fc768ba4e9d59b08b4a20e557b43fa8dc92ae1b" Nov 02 20:52:24.699372 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:24.699349 5800 scope.go:122] "RemoveContainer" containerID="eb19c6649098a2141e6a5d77fe1b0443dc5a2fbf736e45397877e06aec65131a" Nov 02 20:52:24.710816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:24.710795 5800 scope.go:122] "RemoveContainer" containerID="32864fd8396b8025db90ebcdc0ff8317322fb651b6a4f6bdf6bb035f3b3fd4e9" Nov 02 20:52:27.280263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.280225 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7"] Nov 02 20:52:27.280990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.280974 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="146372b4-37e6-4ca6-976b-5f4cae670a6e" containerName="istio-proxy" Nov 02 20:52:27.284631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.284611 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297203 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297246 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297267 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297338 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297488 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297403 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c780e-81ac-4b0e-a286-66724a1e86f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297496 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7wf\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-kube-api-access-gd7wf\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297535 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297599 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297567 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.297652 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.297640 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.340293 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.340244 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\"" Nov 02 20:52:27.343688 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.343663 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7"] Nov 02 20:52:27.398334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398301 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398349 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398474 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398517 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398552 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398545 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c780e-81ac-4b0e-a286-66724a1e86f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398623 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7wf\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-kube-api-access-gd7wf\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398658 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398684 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398734 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398813 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.398864 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.398839 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.399229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.399155 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.399475 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.399310 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.399837 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.399814 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c780e-81ac-4b0e-a286-66724a1e86f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.401299 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.401260 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.401412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.401367 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.455163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.455122 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.456497 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.456470 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7wf\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-kube-api-access-gd7wf\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-b4gn7\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:27.599542 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:27.599454 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:28.030260 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.030220 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7"] Nov 02 20:52:28.073272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.073230 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" podUID="17fedef7-fe57-45be-b0db-cd48a2905442" containerName="payload-processing" containerID="cri-o://44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6" gracePeriod=25 Nov 02 20:52:28.074388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.074346 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:52:28.074524 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.074505 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:52:28.074612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.074595 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:52:28.344138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.344099 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:28.412099 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.412058 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-routing-overlay-volume\") pod \"17fedef7-fe57-45be-b0db-cd48a2905442\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " Nov 02 20:52:28.412335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.412209 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-plugins-config-volume\") pod \"17fedef7-fe57-45be-b0db-cd48a2905442\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " Nov 02 20:52:28.412335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.412249 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-kube-api-access-cw2g9\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-kube-api-access-cw2g9\") pod \"17fedef7-fe57-45be-b0db-cd48a2905442\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " Nov 02 20:52:28.412335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.412274 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-provider-credentials\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-provider-credentials\") pod \"17fedef7-fe57-45be-b0db-cd48a2905442\" (UID: \"17fedef7-fe57-45be-b0db-cd48a2905442\") " Nov 02 20:52:28.412943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.412583 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-routing-overlay-volume" pod "17fedef7-fe57-45be-b0db-cd48a2905442" (UID: "17fedef7-fe57-45be-b0db-cd48a2905442"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:52:28.412943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.412726 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-plugins-config-volume" pod "17fedef7-fe57-45be-b0db-cd48a2905442" (UID: "17fedef7-fe57-45be-b0db-cd48a2905442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:52:28.412943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.412756 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:28.414774 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.414740 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-kube-api-access-cw2g9" pod "17fedef7-fe57-45be-b0db-cd48a2905442" (UID: "17fedef7-fe57-45be-b0db-cd48a2905442"). InnerVolumeSpecName "kube-api-access-cw2g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:28.415239 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.414877 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-provider-credentials" pod "17fedef7-fe57-45be-b0db-cd48a2905442" (UID: "17fedef7-fe57-45be-b0db-cd48a2905442"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:28.514184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.514147 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fedef7-fe57-45be-b0db-cd48a2905442-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:28.514184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.514182 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw2g9\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-kube-api-access-cw2g9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:28.514435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:28.514195 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17fedef7-fe57-45be-b0db-cd48a2905442-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:29.077446 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.077408 5800 generic.go:356] "Generic (PLEG): container finished" podID="17fedef7-fe57-45be-b0db-cd48a2905442" containerID="44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6" exitCode=0 Nov 02 20:52:29.077657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.077473 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" Nov 02 20:52:29.077657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.077484 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" event={"ID":"17fedef7-fe57-45be-b0db-cd48a2905442","Type":"ContainerDied","Data":"44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6"} Nov 02 20:52:29.077657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.077541 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75574cc589-7q5dn" event={"ID":"17fedef7-fe57-45be-b0db-cd48a2905442","Type":"ContainerDied","Data":"4c636f9e03f760b470fb3803da2a87d465d202bf648ebbcc4eabcdb9736855ed"} Nov 02 20:52:29.077657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.077573 5800 scope.go:122] "RemoveContainer" containerID="44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6" Nov 02 20:52:29.086130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.085209 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" event={"ID":"775c780e-81ac-4b0e-a286-66724a1e86f0","Type":"ContainerStarted","Data":"1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386"} Nov 02 20:52:29.086130 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.085268 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" event={"ID":"775c780e-81ac-4b0e-a286-66724a1e86f0","Type":"ContainerStarted","Data":"6c9684dbe04232a51388afa2ef0d37aca3728b7db1ab157a963f2afbf0eeeb67"} Nov 02 20:52:29.097270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.094216 5800 scope.go:122] "RemoveContainer" containerID="44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6" Nov 02 20:52:29.097270 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:52:29.094635 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6\": container with ID starting with 44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6 not found: ID does not exist" containerID="44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6" Nov 02 20:52:29.097270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.094669 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6"} err="failed to get container status \"44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6\": rpc error: code = NotFound desc = could not find container \"44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6\": container with ID starting with 44504ade72f66c8093a59c4249ba1081cea2c0eca5b79c98ca4e89690af934c6 not found: ID does not exist" Nov 02 20:52:29.255828 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.255742 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podStartSLOduration=2.255722635 podStartE2EDuration="2.255722635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:29.25426388 +0000 UTC m=+3847.771036507" watchObservedRunningTime="2026-11-02 20:52:29.255722635 +0000 UTC m=+3847.772495222" Nov 02 20:52:29.363322 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.363226 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75574cc589-7q5dn"] Nov 02 20:52:29.366213 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.366179 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75574cc589-7q5dn"] Nov 02 20:52:29.599609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.599575 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:29.601272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.601246 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 02 20:52:29.601419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.601310 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 02 20:52:29.631354 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:29.625552 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fedef7-fe57-45be-b0db-cd48a2905442" path="/var/lib/kubelet/pods/17fedef7-fe57-45be-b0db-cd48a2905442/volumes" Nov 02 20:52:30.600570 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:30.600523 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 02 20:52:30.601013 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:30.600593 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 02 20:52:31.599909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:31.599876 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 02 20:52:31.600091 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:31.599927 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 02 20:52:32.600836 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:32.600797 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 02 20:52:32.601263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:32.600866 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 02 20:52:33.599897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:33.599859 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 02 20:52:33.600099 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:33.599918 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 02 20:52:34.600714 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:34.600668 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 02 20:52:34.601143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:34.600736 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 02 20:52:35.600108 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:35.600068 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 02 20:52:35.600315 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:35.600130 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 02 20:52:37.104237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:37.104204 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:37.104822 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:37.104791 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:37.106007 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:37.105989 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 20:52:52.697582 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:52:52.697539 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:52:55.882028 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.881986 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc"] Nov 02 20:52:55.883020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.882996 5800 cpu_manager.go:414] "containerMap: removing container" podUID="17fedef7-fe57-45be-b0db-cd48a2905442" containerName="payload-processing" Nov 02 20:52:55.883070 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.883024 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fedef7-fe57-45be-b0db-cd48a2905442" containerName="payload-processing" Nov 02 20:52:55.883164 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.883153 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fedef7-fe57-45be-b0db-cd48a2905442" containerName="payload-processing" Nov 02 20:52:55.887266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.887246 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:55.937625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.937592 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc"] Nov 02 20:52:55.979858 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.979824 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:52:55.980069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.979897 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:52:55.980069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.979920 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:52:55.982767 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.982744 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:52:55.990476 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.990449 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:55.990625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.990485 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:55.990752 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:55.990725 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6d2\" (UniqueName: \"kubernetes.io/projected/def9a282-bf2a-4dc2-9687-03d42f0a4a41-kube-api-access-cd6d2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.091473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.091436 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.091473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.091475 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.091694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.091570 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6d2\" (UniqueName: \"kubernetes.io/projected/def9a282-bf2a-4dc2-9687-03d42f0a4a41-kube-api-access-cd6d2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.092185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.092161 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.093877 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.093854 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.144390 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.144326 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6d2\" (UniqueName: \"kubernetes.io/projected/def9a282-bf2a-4dc2-9687-03d42f0a4a41-kube-api-access-cd6d2\") pod \"maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.202297 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.202259 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:56.569973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:56.569945 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc"] Nov 02 20:52:56.571883 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:52:56.571847 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef9a282_bf2a_4dc2_9687_03d42f0a4a41.slice/crio-35c59183ffdd46e324c417c75cd5668634770d05cfddc20b78c3359167870512 WatchSource:0}: Error finding container 35c59183ffdd46e324c417c75cd5668634770d05cfddc20b78c3359167870512: Status 404 returned error can't find the container with id 35c59183ffdd46e324c417c75cd5668634770d05cfddc20b78c3359167870512 Nov 02 20:52:57.307096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:57.307061 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" event={"ID":"def9a282-bf2a-4dc2-9687-03d42f0a4a41","Type":"ContainerStarted","Data":"0ed3e7df7b6ee5092871ea02190eb6417d605f58ea686ca01c40bc66e2f9b043"} Nov 02 20:52:57.307640 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:57.307115 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" event={"ID":"def9a282-bf2a-4dc2-9687-03d42f0a4a41","Type":"ContainerStarted","Data":"35c59183ffdd46e324c417c75cd5668634770d05cfddc20b78c3359167870512"} Nov 02 20:52:57.412720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:57.412663 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" podStartSLOduration=2.412648239 podStartE2EDuration="2.412648239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:52:57.412039961 +0000 UTC m=+3875.928812546" watchObservedRunningTime="2026-11-02 20:52:57.412648239 +0000 UTC m=+3875.929420824" Nov 02 20:52:58.317464 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:58.317431 5800 generic.go:356] "Generic (PLEG): container finished" podID="def9a282-bf2a-4dc2-9687-03d42f0a4a41" containerID="0ed3e7df7b6ee5092871ea02190eb6417d605f58ea686ca01c40bc66e2f9b043" exitCode=0 Nov 02 20:52:58.318023 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:58.317515 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" event={"ID":"def9a282-bf2a-4dc2-9687-03d42f0a4a41","Type":"ContainerDied","Data":"0ed3e7df7b6ee5092871ea02190eb6417d605f58ea686ca01c40bc66e2f9b043"} Nov 02 20:52:59.477694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.477667 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:52:59.539177 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.539144 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-openshift-ca\") pod \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " Nov 02 20:52:59.539410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.539234 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def9a282-bf2a-4dc2-9687-03d42f0a4a41-kube-api-access-cd6d2\" (UniqueName: \"kubernetes.io/projected/def9a282-bf2a-4dc2-9687-03d42f0a4a41-kube-api-access-cd6d2\") pod \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " Nov 02 20:52:59.539410 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.539260 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-xks-ca\") pod \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\" (UID: \"def9a282-bf2a-4dc2-9687-03d42f0a4a41\") " Nov 02 20:52:59.539559 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.539531 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-openshift-ca" pod "def9a282-bf2a-4dc2-9687-03d42f0a4a41" (UID: "def9a282-bf2a-4dc2-9687-03d42f0a4a41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:52:59.539818 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.539793 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:59.541423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.541405 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-xks-ca" pod "def9a282-bf2a-4dc2-9687-03d42f0a4a41" (UID: "def9a282-bf2a-4dc2-9687-03d42f0a4a41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:52:59.541540 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.541419 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def9a282-bf2a-4dc2-9687-03d42f0a4a41-kube-api-access-cd6d2" pod "def9a282-bf2a-4dc2-9687-03d42f0a4a41" (UID: "def9a282-bf2a-4dc2-9687-03d42f0a4a41"). InnerVolumeSpecName "kube-api-access-cd6d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:59.641512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.641484 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6d2\" (UniqueName: \"kubernetes.io/projected/def9a282-bf2a-4dc2-9687-03d42f0a4a41-kube-api-access-cd6d2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:59.641512 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:52:59.641512 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def9a282-bf2a-4dc2-9687-03d42f0a4a41-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:00.334356 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:00.334327 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" Nov 02 20:53:00.334534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:00.334328 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc" event={"ID":"def9a282-bf2a-4dc2-9687-03d42f0a4a41","Type":"ContainerDied","Data":"35c59183ffdd46e324c417c75cd5668634770d05cfddc20b78c3359167870512"} Nov 02 20:53:00.334534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:00.334483 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c59183ffdd46e324c417c75cd5668634770d05cfddc20b78c3359167870512" Nov 02 20:53:10.881421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:10.881387 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc"] Nov 02 20:53:10.980945 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:10.980910 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb"] Nov 02 20:53:15.882388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:15.882340 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" podUID="307b9e36-2241-4431-84de-cc009ad00f35" containerName="payload-processing" containerID="cri-o://a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86" gracePeriod=25 Nov 02 20:53:15.982211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:15.982158 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" podUID="2a14cd55-526a-4d37-aaaf-e865adaca4f4" containerName="payload-pre-processing" containerID="cri-o://ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f" gracePeriod=25 Nov 02 20:53:16.187931 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.187892 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:53:16.254275 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.254253 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:53:16.274592 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.274564 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307b9e36-2241-4431-84de-cc009ad00f35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307b9e36-2241-4431-84de-cc009ad00f35-plugins-config-volume\") pod \"307b9e36-2241-4431-84de-cc009ad00f35\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " Nov 02 20:53:16.274726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.274651 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307b9e36-2241-4431-84de-cc009ad00f35-kube-api-access-hcjlm\" (UniqueName: \"kubernetes.io/projected/307b9e36-2241-4431-84de-cc009ad00f35-kube-api-access-hcjlm\") pod \"307b9e36-2241-4431-84de-cc009ad00f35\" (UID: \"307b9e36-2241-4431-84de-cc009ad00f35\") " Nov 02 20:53:16.274808 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.274792 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a14cd55-526a-4d37-aaaf-e865adaca4f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a14cd55-526a-4d37-aaaf-e865adaca4f4-plugins-config-volume\") pod \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " Nov 02 20:53:16.274928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.274911 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a14cd55-526a-4d37-aaaf-e865adaca4f4-kube-api-access-9jxvg\" (UniqueName: \"kubernetes.io/projected/2a14cd55-526a-4d37-aaaf-e865adaca4f4-kube-api-access-9jxvg\") pod \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\" (UID: \"2a14cd55-526a-4d37-aaaf-e865adaca4f4\") " Nov 02 20:53:16.275017 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.274986 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307b9e36-2241-4431-84de-cc009ad00f35-plugins-config-volume" pod "307b9e36-2241-4431-84de-cc009ad00f35" (UID: "307b9e36-2241-4431-84de-cc009ad00f35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:16.275200 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.275178 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a14cd55-526a-4d37-aaaf-e865adaca4f4-plugins-config-volume" pod "2a14cd55-526a-4d37-aaaf-e865adaca4f4" (UID: "2a14cd55-526a-4d37-aaaf-e865adaca4f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:16.275436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.275418 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a14cd55-526a-4d37-aaaf-e865adaca4f4-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:16.275529 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.275438 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/307b9e36-2241-4431-84de-cc009ad00f35-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:16.276856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.276835 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307b9e36-2241-4431-84de-cc009ad00f35-kube-api-access-hcjlm" pod "307b9e36-2241-4431-84de-cc009ad00f35" (UID: "307b9e36-2241-4431-84de-cc009ad00f35"). InnerVolumeSpecName "kube-api-access-hcjlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:16.277168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.277146 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a14cd55-526a-4d37-aaaf-e865adaca4f4-kube-api-access-9jxvg" pod "2a14cd55-526a-4d37-aaaf-e865adaca4f4" (UID: "2a14cd55-526a-4d37-aaaf-e865adaca4f4"). InnerVolumeSpecName "kube-api-access-9jxvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:16.376394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.376359 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjlm\" (UniqueName: \"kubernetes.io/projected/307b9e36-2241-4431-84de-cc009ad00f35-kube-api-access-hcjlm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:16.376394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.376387 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxvg\" (UniqueName: \"kubernetes.io/projected/2a14cd55-526a-4d37-aaaf-e865adaca4f4-kube-api-access-9jxvg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:16.473942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.473854 5800 generic.go:356] "Generic (PLEG): container finished" podID="307b9e36-2241-4431-84de-cc009ad00f35" containerID="a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86" exitCode=0 Nov 02 20:53:16.473942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.473895 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" Nov 02 20:53:16.473942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.473929 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" event={"ID":"307b9e36-2241-4431-84de-cc009ad00f35","Type":"ContainerDied","Data":"a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86"} Nov 02 20:53:16.474295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.473985 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc" event={"ID":"307b9e36-2241-4431-84de-cc009ad00f35","Type":"ContainerDied","Data":"5ab32f559d140201709d837317a3326d3fc09c7640bcc29a21751f39df10a773"} Nov 02 20:53:16.474295 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.474004 5800 scope.go:122] "RemoveContainer" containerID="a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86" Nov 02 20:53:16.475600 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.475578 5800 generic.go:356] "Generic (PLEG): container finished" podID="2a14cd55-526a-4d37-aaaf-e865adaca4f4" containerID="ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f" exitCode=0 Nov 02 20:53:16.475720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.475630 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" Nov 02 20:53:16.475720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.475683 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" event={"ID":"2a14cd55-526a-4d37-aaaf-e865adaca4f4","Type":"ContainerDied","Data":"ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f"} Nov 02 20:53:16.475847 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.475738 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb" event={"ID":"2a14cd55-526a-4d37-aaaf-e865adaca4f4","Type":"ContainerDied","Data":"1e72ed9d049a4e074c0ee6ce3867f21ff48889886c67b38685450b3493ae73b7"} Nov 02 20:53:16.486137 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.486117 5800 scope.go:122] "RemoveContainer" containerID="a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86" Nov 02 20:53:16.486433 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:53:16.486414 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86\": container with ID starting with a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86 not found: ID does not exist" containerID="a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86" Nov 02 20:53:16.486511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.486445 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86"} err="failed to get container status \"a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86\": rpc error: code = NotFound desc = could not find container \"a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86\": container with ID starting with a17e20d7e3af4af8cdff0672dea4713b6c1ca841156b42f5c81704098eb5cb86 not found: ID does not exist" Nov 02 20:53:16.486511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.486464 5800 scope.go:122] "RemoveContainer" containerID="ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f" Nov 02 20:53:16.497175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.497157 5800 scope.go:122] "RemoveContainer" containerID="ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f" Nov 02 20:53:16.497448 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:53:16.497426 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f\": container with ID starting with ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f not found: ID does not exist" containerID="ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f" Nov 02 20:53:16.497504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.497456 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f"} err="failed to get container status \"ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f\": rpc error: code = NotFound desc = could not find container \"ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f\": container with ID starting with ddb7e314df603feafc23844e7afa3037b3080a328933dedb5468e540403c0f8f not found: ID does not exist" Nov 02 20:53:16.628807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.628776 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc"] Nov 02 20:53:16.633684 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.633657 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-27e421-54779d65cf-tbstc"] Nov 02 20:53:16.732511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.732422 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb"] Nov 02 20:53:16.737163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:16.737132 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-27e421-65b9995cf9-92szb"] Nov 02 20:53:17.626079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:17.626039 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a14cd55-526a-4d37-aaaf-e865adaca4f4" path="/var/lib/kubelet/pods/2a14cd55-526a-4d37-aaaf-e865adaca4f4/volumes" Nov 02 20:53:17.626682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:17.626660 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307b9e36-2241-4431-84de-cc009ad00f35" path="/var/lib/kubelet/pods/307b9e36-2241-4431-84de-cc009ad00f35/volumes" Nov 02 20:53:22.376567 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:22.376533 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:53:22.379526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:22.379498 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:53:22.701897 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:53:22.701808 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:53:35.146154 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:35.146115 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n"] Nov 02 20:53:35.247682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:35.247646 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t"] Nov 02 20:53:40.147266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.147224 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" podUID="530b676f-60cd-4e6c-8be0-4bc12aa39870" containerName="payload-processing" containerID="cri-o://e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e" gracePeriod=25 Nov 02 20:53:40.248634 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.248589 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" podUID="c323daa6-73b4-45eb-9314-71566a4f2652" containerName="payload-pre-processing" containerID="cri-o://a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5" gracePeriod=25 Nov 02 20:53:40.420612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.420551 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:53:40.539092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.539061 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:53:40.606057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.606024 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530b676f-60cd-4e6c-8be0-4bc12aa39870-kube-api-access-hp5tj\" (UniqueName: \"kubernetes.io/projected/530b676f-60cd-4e6c-8be0-4bc12aa39870-kube-api-access-hp5tj\") pod \"530b676f-60cd-4e6c-8be0-4bc12aa39870\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " Nov 02 20:53:40.606246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.606212 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530b676f-60cd-4e6c-8be0-4bc12aa39870-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530b676f-60cd-4e6c-8be0-4bc12aa39870-plugins-config-volume\") pod \"530b676f-60cd-4e6c-8be0-4bc12aa39870\" (UID: \"530b676f-60cd-4e6c-8be0-4bc12aa39870\") " Nov 02 20:53:40.606612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.606589 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530b676f-60cd-4e6c-8be0-4bc12aa39870-plugins-config-volume" pod "530b676f-60cd-4e6c-8be0-4bc12aa39870" (UID: "530b676f-60cd-4e6c-8be0-4bc12aa39870"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:40.606753 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.606738 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530b676f-60cd-4e6c-8be0-4bc12aa39870-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:40.608180 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.608156 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530b676f-60cd-4e6c-8be0-4bc12aa39870-kube-api-access-hp5tj" pod "530b676f-60cd-4e6c-8be0-4bc12aa39870" (UID: "530b676f-60cd-4e6c-8be0-4bc12aa39870"). InnerVolumeSpecName "kube-api-access-hp5tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:40.708059 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.707969 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c323daa6-73b4-45eb-9314-71566a4f2652-kube-api-access-jtvhw\" (UniqueName: \"kubernetes.io/projected/c323daa6-73b4-45eb-9314-71566a4f2652-kube-api-access-jtvhw\") pod \"c323daa6-73b4-45eb-9314-71566a4f2652\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " Nov 02 20:53:40.708238 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.708072 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c323daa6-73b4-45eb-9314-71566a4f2652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c323daa6-73b4-45eb-9314-71566a4f2652-plugins-config-volume\") pod \"c323daa6-73b4-45eb-9314-71566a4f2652\" (UID: \"c323daa6-73b4-45eb-9314-71566a4f2652\") " Nov 02 20:53:40.708491 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.708458 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c323daa6-73b4-45eb-9314-71566a4f2652-plugins-config-volume" pod "c323daa6-73b4-45eb-9314-71566a4f2652" (UID: "c323daa6-73b4-45eb-9314-71566a4f2652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:40.708624 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.708612 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5tj\" (UniqueName: \"kubernetes.io/projected/530b676f-60cd-4e6c-8be0-4bc12aa39870-kube-api-access-hp5tj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:40.708687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.708624 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c323daa6-73b4-45eb-9314-71566a4f2652-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:40.710187 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.710166 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c323daa6-73b4-45eb-9314-71566a4f2652-kube-api-access-jtvhw" pod "c323daa6-73b4-45eb-9314-71566a4f2652" (UID: "c323daa6-73b4-45eb-9314-71566a4f2652"). InnerVolumeSpecName "kube-api-access-jtvhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:40.733545 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.733509 5800 generic.go:356] "Generic (PLEG): container finished" podID="c323daa6-73b4-45eb-9314-71566a4f2652" containerID="a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5" exitCode=0 Nov 02 20:53:40.733726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.733564 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" Nov 02 20:53:40.733726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.733601 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" event={"ID":"c323daa6-73b4-45eb-9314-71566a4f2652","Type":"ContainerDied","Data":"a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5"} Nov 02 20:53:40.733726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.733666 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t" event={"ID":"c323daa6-73b4-45eb-9314-71566a4f2652","Type":"ContainerDied","Data":"eb618f121d5ae0a4803fe4c95d3e457f6b00c498862aed62d884aab762d722cc"} Nov 02 20:53:40.733726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.733699 5800 scope.go:122] "RemoveContainer" containerID="a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5" Nov 02 20:53:40.735642 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.735616 5800 generic.go:356] "Generic (PLEG): container finished" podID="530b676f-60cd-4e6c-8be0-4bc12aa39870" containerID="e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e" exitCode=0 Nov 02 20:53:40.735746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.735667 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" Nov 02 20:53:40.735746 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.735711 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" event={"ID":"530b676f-60cd-4e6c-8be0-4bc12aa39870","Type":"ContainerDied","Data":"e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e"} Nov 02 20:53:40.735868 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.735796 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n" event={"ID":"530b676f-60cd-4e6c-8be0-4bc12aa39870","Type":"ContainerDied","Data":"c42178a884abbcb14eae198ee0a7b194c246cfa2146e1e020882d7fc9a5302f8"} Nov 02 20:53:40.746963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.746937 5800 scope.go:122] "RemoveContainer" containerID="a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5" Nov 02 20:53:40.747352 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:53:40.747324 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5\": container with ID starting with a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5 not found: ID does not exist" containerID="a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5" Nov 02 20:53:40.747454 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.747364 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5"} err="failed to get container status \"a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5\": rpc error: code = NotFound desc = could not find container \"a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5\": container with ID starting with a3ade87a304d1ff3ad1dd1533d3d0d62ba9a0359fd85ab4d84096c88c74381a5 not found: ID does not exist" Nov 02 20:53:40.747454 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.747398 5800 scope.go:122] "RemoveContainer" containerID="e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e" Nov 02 20:53:40.757205 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.757186 5800 scope.go:122] "RemoveContainer" containerID="e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e" Nov 02 20:53:40.757555 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:53:40.757538 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e\": container with ID starting with e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e not found: ID does not exist" containerID="e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e" Nov 02 20:53:40.757615 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.757562 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e"} err="failed to get container status \"e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e\": rpc error: code = NotFound desc = could not find container \"e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e\": container with ID starting with e5a72b7e5a7074c4fa0d3c70d22b914ac4b7b3b48ac6a1291e333c49a2f2057e not found: ID does not exist" Nov 02 20:53:40.809789 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.809757 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtvhw\" (UniqueName: \"kubernetes.io/projected/c323daa6-73b4-45eb-9314-71566a4f2652-kube-api-access-jtvhw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:40.887675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.887641 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n"] Nov 02 20:53:40.892563 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.892530 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-28197c-7d6557d4f9-gkx2n"] Nov 02 20:53:40.994423 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.994343 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t"] Nov 02 20:53:40.997121 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:40.997097 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-28197c-7b7d4d7c95-p262t"] Nov 02 20:53:41.094171 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:41.094137 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c"] Nov 02 20:53:41.624531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:41.624502 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530b676f-60cd-4e6c-8be0-4bc12aa39870" path="/var/lib/kubelet/pods/530b676f-60cd-4e6c-8be0-4bc12aa39870/volumes" Nov 02 20:53:41.624924 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:41.624893 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c323daa6-73b4-45eb-9314-71566a4f2652" path="/var/lib/kubelet/pods/c323daa6-73b4-45eb-9314-71566a4f2652/volumes" Nov 02 20:53:46.094999 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.094946 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" podUID="739dc8e6-1fac-45bb-b38f-38b745c0f627" containerName="payload-processing" containerID="cri-o://2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83" gracePeriod=25 Nov 02 20:53:46.388047 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.388016 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:53:46.489713 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.489676 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-routing-overlay-volume\") pod \"739dc8e6-1fac-45bb-b38f-38b745c0f627\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " Nov 02 20:53:46.489881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.489798 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-plugins-config-volume\") pod \"739dc8e6-1fac-45bb-b38f-38b745c0f627\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " Nov 02 20:53:46.489881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.489845 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-kube-api-access-dntvh\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-kube-api-access-dntvh\") pod \"739dc8e6-1fac-45bb-b38f-38b745c0f627\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " Nov 02 20:53:46.489881 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.489871 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-provider-credentials\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-provider-credentials\") pod \"739dc8e6-1fac-45bb-b38f-38b745c0f627\" (UID: \"739dc8e6-1fac-45bb-b38f-38b745c0f627\") " Nov 02 20:53:46.490163 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.490138 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-routing-overlay-volume" pod "739dc8e6-1fac-45bb-b38f-38b745c0f627" (UID: "739dc8e6-1fac-45bb-b38f-38b745c0f627"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:46.490335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.490272 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-plugins-config-volume" pod "739dc8e6-1fac-45bb-b38f-38b745c0f627" (UID: "739dc8e6-1fac-45bb-b38f-38b745c0f627"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:53:46.490462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.490379 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:46.492138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.492105 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-provider-credentials" pod "739dc8e6-1fac-45bb-b38f-38b745c0f627" (UID: "739dc8e6-1fac-45bb-b38f-38b745c0f627"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:46.492215 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.492146 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-kube-api-access-dntvh" pod "739dc8e6-1fac-45bb-b38f-38b745c0f627" (UID: "739dc8e6-1fac-45bb-b38f-38b745c0f627"). InnerVolumeSpecName "kube-api-access-dntvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:53:46.591242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.591201 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739dc8e6-1fac-45bb-b38f-38b745c0f627-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:46.591242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.591235 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dntvh\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-kube-api-access-dntvh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:46.591242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.591250 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/739dc8e6-1fac-45bb-b38f-38b745c0f627-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:53:46.782657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.782618 5800 generic.go:356] "Generic (PLEG): container finished" podID="739dc8e6-1fac-45bb-b38f-38b745c0f627" containerID="2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83" exitCode=0 Nov 02 20:53:46.782952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.782659 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" Nov 02 20:53:46.782952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.782690 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" event={"ID":"739dc8e6-1fac-45bb-b38f-38b745c0f627","Type":"ContainerDied","Data":"2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83"} Nov 02 20:53:46.782952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.782741 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c" event={"ID":"739dc8e6-1fac-45bb-b38f-38b745c0f627","Type":"ContainerDied","Data":"78dd3b9187321b3cee397b8638d2c7823e16cf618a777df83c90d26fd6faf0df"} Nov 02 20:53:46.782952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.782761 5800 scope.go:122] "RemoveContainer" containerID="2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83" Nov 02 20:53:46.795190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.795172 5800 scope.go:122] "RemoveContainer" containerID="2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83" Nov 02 20:53:46.795490 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:53:46.795467 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83\": container with ID starting with 2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83 not found: ID does not exist" containerID="2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83" Nov 02 20:53:46.795579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.795496 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83"} err="failed to get container status \"2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83\": rpc error: code = NotFound desc = could not find container \"2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83\": container with ID starting with 2a60ed00cb25ef8e19d1a85b37961060e90af308501400427396d0f3a6223b83 not found: ID does not exist" Nov 02 20:53:46.937543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.937508 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c"] Nov 02 20:53:46.940972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:46.940948 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-544bc8ddf6-fdv4c"] Nov 02 20:53:47.623996 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:53:47.623965 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739dc8e6-1fac-45bb-b38f-38b745c0f627" path="/var/lib/kubelet/pods/739dc8e6-1fac-45bb-b38f-38b745c0f627/volumes" Nov 02 20:53:52.696124 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:53:52.696089 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:54:04.152346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.152309 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc"] Nov 02 20:54:04.152922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.152896 5800 cpu_manager.go:414] "containerMap: removing container" podUID="530b676f-60cd-4e6c-8be0-4bc12aa39870" containerName="payload-processing" Nov 02 20:54:04.152922 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.152916 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530b676f-60cd-4e6c-8be0-4bc12aa39870" containerName="payload-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.152951 5800 cpu_manager.go:414] "containerMap: removing container" podUID="739dc8e6-1fac-45bb-b38f-38b745c0f627" containerName="payload-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.152958 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739dc8e6-1fac-45bb-b38f-38b745c0f627" containerName="payload-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.152973 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2a14cd55-526a-4d37-aaaf-e865adaca4f4" containerName="payload-pre-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.152980 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a14cd55-526a-4d37-aaaf-e865adaca4f4" containerName="payload-pre-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153009 5800 cpu_manager.go:414] "containerMap: removing container" podUID="307b9e36-2241-4431-84de-cc009ad00f35" containerName="payload-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153019 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307b9e36-2241-4431-84de-cc009ad00f35" containerName="payload-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153039 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c323daa6-73b4-45eb-9314-71566a4f2652" containerName="payload-pre-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153046 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c323daa6-73b4-45eb-9314-71566a4f2652" containerName="payload-pre-processing" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153057 5800 cpu_manager.go:414] "containerMap: removing container" podUID="def9a282-bf2a-4dc2-9687-03d42f0a4a41" containerName="revoke-keys" Nov 02 20:54:04.153125 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153065 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def9a282-bf2a-4dc2-9687-03d42f0a4a41" containerName="revoke-keys" Nov 02 20:54:04.153693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153225 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c323daa6-73b4-45eb-9314-71566a4f2652" containerName="payload-pre-processing" Nov 02 20:54:04.153693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153238 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="def9a282-bf2a-4dc2-9687-03d42f0a4a41" containerName="revoke-keys" Nov 02 20:54:04.153693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153254 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="530b676f-60cd-4e6c-8be0-4bc12aa39870" containerName="payload-processing" Nov 02 20:54:04.153693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153269 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="739dc8e6-1fac-45bb-b38f-38b745c0f627" containerName="payload-processing" Nov 02 20:54:04.153693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153296 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a14cd55-526a-4d37-aaaf-e865adaca4f4" containerName="payload-pre-processing" Nov 02 20:54:04.153693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.153309 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="307b9e36-2241-4431-84de-cc009ad00f35" containerName="payload-processing" Nov 02 20:54:04.156963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.156943 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.208938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.208904 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc"] Nov 02 20:54:04.251156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.251117 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:54:04.251156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.251148 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:54:04.251396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.251136 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:54:04.252462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.252442 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 20:54:04.296334 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.296273 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.296492 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.296456 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.296592 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.296568 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58v9b\" (UniqueName: \"kubernetes.io/projected/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-kube-api-access-58v9b\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.397582 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.397546 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.397807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.397630 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58v9b\" (UniqueName: \"kubernetes.io/projected/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-kube-api-access-58v9b\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.397807 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.397673 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.398348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.398323 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.399958 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.399937 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.451090 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.451018 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58v9b\" (UniqueName: \"kubernetes.io/projected/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-kube-api-access-58v9b\") pod \"maas-api-revoke-sub-ae129e97-pqnfc\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.481940 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.481902 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:04.882046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.882012 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc"] Nov 02 20:54:04.882877 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:54:04.882851 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcbe8fd_e744_4877_9c8c_7462dac7bd2f.slice/crio-638bb30cddeb6f134f3c1e4cc1dc22bd30de8736c2112d9ccfa9c4f76e8c932f WatchSource:0}: Error finding container 638bb30cddeb6f134f3c1e4cc1dc22bd30de8736c2112d9ccfa9c4f76e8c932f: Status 404 returned error can't find the container with id 638bb30cddeb6f134f3c1e4cc1dc22bd30de8736c2112d9ccfa9c4f76e8c932f Nov 02 20:54:04.926994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:04.926962 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" event={"ID":"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f","Type":"ContainerStarted","Data":"638bb30cddeb6f134f3c1e4cc1dc22bd30de8736c2112d9ccfa9c4f76e8c932f"} Nov 02 20:54:05.936246 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:05.936214 5800 generic.go:356] "Generic (PLEG): container finished" podID="5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" containerID="e13dc62914db49202d7d3e7ccec3b71d996e1322cbdd66927ca6049291d98efa" exitCode=0 Nov 02 20:54:05.936694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:05.936322 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" event={"ID":"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f","Type":"ContainerDied","Data":"e13dc62914db49202d7d3e7ccec3b71d996e1322cbdd66927ca6049291d98efa"} Nov 02 20:54:06.226358 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.226260 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5"] Nov 02 20:54:06.231979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.231959 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.281361 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.281334 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5"] Nov 02 20:54:06.429952 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.429918 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85020637-1049-484f-9c04-f402db4a8e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.430161 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.429976 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8j2\" (UniqueName: \"kubernetes.io/projected/85020637-1049-484f-9c04-f402db4a8e58-kube-api-access-cm8j2\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.430237 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.430204 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85020637-1049-484f-9c04-f402db4a8e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.531850 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.531811 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8j2\" (UniqueName: \"kubernetes.io/projected/85020637-1049-484f-9c04-f402db4a8e58-kube-api-access-cm8j2\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.532071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.531970 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85020637-1049-484f-9c04-f402db4a8e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.532144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.532102 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85020637-1049-484f-9c04-f402db4a8e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.532856 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.532828 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85020637-1049-484f-9c04-f402db4a8e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.534548 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.534520 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85020637-1049-484f-9c04-f402db4a8e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.584589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.584551 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8j2\" (UniqueName: \"kubernetes.io/projected/85020637-1049-484f-9c04-f402db4a8e58-kube-api-access-cm8j2\") pod \"maas-api-revoke-sub-0e8738b1-f7xr5\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:06.846406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:06.846314 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:07.096122 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.096100 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:07.140096 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.140010 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-kube-api-access-58v9b\" (UniqueName: \"kubernetes.io/projected/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-kube-api-access-58v9b\") pod \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " Nov 02 20:54:07.140276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.140149 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-xks-ca\") pod \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " Nov 02 20:54:07.140276 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.140193 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-openshift-ca\") pod \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\" (UID: \"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f\") " Nov 02 20:54:07.140665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.140633 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-openshift-ca" pod "5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" (UID: "5bcbe8fd-e744-4877-9c8c-7462dac7bd2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:07.140797 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.140779 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:07.142229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.142206 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-kube-api-access-58v9b" pod "5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" (UID: "5bcbe8fd-e744-4877-9c8c-7462dac7bd2f"). InnerVolumeSpecName "kube-api-access-58v9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:07.142398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.142347 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-xks-ca" pod "5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" (UID: "5bcbe8fd-e744-4877-9c8c-7462dac7bd2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:07.219855 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:54:07.219818 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85020637_1049_484f_9c04_f402db4a8e58.slice/crio-792fe9985f2c2e1c20175bf9978ad1f81981c4d5befb5435d1dffb004e98411e WatchSource:0}: Error finding container 792fe9985f2c2e1c20175bf9978ad1f81981c4d5befb5435d1dffb004e98411e: Status 404 returned error can't find the container with id 792fe9985f2c2e1c20175bf9978ad1f81981c4d5befb5435d1dffb004e98411e Nov 02 20:54:07.241764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.241737 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:07.241764 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.241766 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58v9b\" (UniqueName: \"kubernetes.io/projected/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f-kube-api-access-58v9b\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:07.251785 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.251761 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5"] Nov 02 20:54:07.958232 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.958196 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" Nov 02 20:54:07.958430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.958228 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc" event={"ID":"5bcbe8fd-e744-4877-9c8c-7462dac7bd2f","Type":"ContainerDied","Data":"638bb30cddeb6f134f3c1e4cc1dc22bd30de8736c2112d9ccfa9c4f76e8c932f"} Nov 02 20:54:07.958430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.958305 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638bb30cddeb6f134f3c1e4cc1dc22bd30de8736c2112d9ccfa9c4f76e8c932f" Nov 02 20:54:07.960141 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.960111 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" event={"ID":"85020637-1049-484f-9c04-f402db4a8e58","Type":"ContainerStarted","Data":"fbb2f73a87b18ea78feafbbb378125785c05b9b8c18f56668294de9f2d0d5856"} Nov 02 20:54:07.960292 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:07.960159 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" event={"ID":"85020637-1049-484f-9c04-f402db4a8e58","Type":"ContainerStarted","Data":"792fe9985f2c2e1c20175bf9978ad1f81981c4d5befb5435d1dffb004e98411e"} Nov 02 20:54:08.066093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:08.066031 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" podStartSLOduration=2.066017473 podStartE2EDuration="2.066017473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:08.064950165 +0000 UTC m=+3946.581722751" watchObservedRunningTime="2026-11-02 20:54:08.066017473 +0000 UTC m=+3946.582790058" Nov 02 20:54:08.969758 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:08.969721 5800 generic.go:356] "Generic (PLEG): container finished" podID="85020637-1049-484f-9c04-f402db4a8e58" containerID="fbb2f73a87b18ea78feafbbb378125785c05b9b8c18f56668294de9f2d0d5856" exitCode=0 Nov 02 20:54:08.970176 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:08.969809 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" event={"ID":"85020637-1049-484f-9c04-f402db4a8e58","Type":"ContainerDied","Data":"fbb2f73a87b18ea78feafbbb378125785c05b9b8c18f56668294de9f2d0d5856"} Nov 02 20:54:10.143434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.143407 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:10.183694 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.183662 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85020637-1049-484f-9c04-f402db4a8e58-kube-api-access-cm8j2\" (UniqueName: \"kubernetes.io/projected/85020637-1049-484f-9c04-f402db4a8e58-kube-api-access-cm8j2\") pod \"85020637-1049-484f-9c04-f402db4a8e58\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " Nov 02 20:54:10.183950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.183732 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85020637-1049-484f-9c04-f402db4a8e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85020637-1049-484f-9c04-f402db4a8e58-maas-api-xks-ca\") pod \"85020637-1049-484f-9c04-f402db4a8e58\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " Nov 02 20:54:10.183950 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.183906 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85020637-1049-484f-9c04-f402db4a8e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85020637-1049-484f-9c04-f402db4a8e58-maas-api-openshift-ca\") pod \"85020637-1049-484f-9c04-f402db4a8e58\" (UID: \"85020637-1049-484f-9c04-f402db4a8e58\") " Nov 02 20:54:10.184594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.184553 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85020637-1049-484f-9c04-f402db4a8e58-maas-api-openshift-ca" pod "85020637-1049-484f-9c04-f402db4a8e58" (UID: "85020637-1049-484f-9c04-f402db4a8e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:10.186098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.186057 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85020637-1049-484f-9c04-f402db4a8e58-kube-api-access-cm8j2" pod "85020637-1049-484f-9c04-f402db4a8e58" (UID: "85020637-1049-484f-9c04-f402db4a8e58"). InnerVolumeSpecName "kube-api-access-cm8j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:10.186098 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.186086 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85020637-1049-484f-9c04-f402db4a8e58-maas-api-xks-ca" pod "85020637-1049-484f-9c04-f402db4a8e58" (UID: "85020637-1049-484f-9c04-f402db4a8e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:54:10.285866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.285825 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85020637-1049-484f-9c04-f402db4a8e58-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:10.285866 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.285866 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8j2\" (UniqueName: \"kubernetes.io/projected/85020637-1049-484f-9c04-f402db4a8e58-kube-api-access-cm8j2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:10.286099 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.285882 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85020637-1049-484f-9c04-f402db4a8e58-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:10.985436 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.985402 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" Nov 02 20:54:10.985660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.985436 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5" event={"ID":"85020637-1049-484f-9c04-f402db4a8e58","Type":"ContainerDied","Data":"792fe9985f2c2e1c20175bf9978ad1f81981c4d5befb5435d1dffb004e98411e"} Nov 02 20:54:10.985660 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:10.985484 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="792fe9985f2c2e1c20175bf9978ad1f81981c4d5befb5435d1dffb004e98411e" Nov 02 20:54:17.376057 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.376019 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w"] Nov 02 20:54:17.378541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.376617 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" containerName="revoke-keys" Nov 02 20:54:17.378541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.376629 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" containerName="revoke-keys" Nov 02 20:54:17.378541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.376661 5800 cpu_manager.go:414] "containerMap: removing container" podUID="85020637-1049-484f-9c04-f402db4a8e58" containerName="revoke-keys" Nov 02 20:54:17.378541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.376666 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85020637-1049-484f-9c04-f402db4a8e58" containerName="revoke-keys" Nov 02 20:54:17.378541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.376767 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" containerName="revoke-keys" Nov 02 20:54:17.378541 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.376781 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="85020637-1049-484f-9c04-f402db4a8e58" containerName="revoke-keys" Nov 02 20:54:17.380135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.380112 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.476216 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.476170 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8a438af1b0c\"" Nov 02 20:54:17.476483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.476276 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 20:54:17.476483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.476352 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c233618dc3e5\"" Nov 02 20:54:17.476483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.476368 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 20:54:17.476483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.476392 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lkhrv\"" Nov 02 20:54:17.476483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.476417 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 20:54:17.476483 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.476452 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 20:54:17.480056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.480037 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w"] Nov 02 20:54:17.481609 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.481591 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbrk\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-kube-api-access-cwbrk\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.481679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.481651 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.481829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.481807 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-routing-overlay-volume\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.481897 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.481855 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-plugins-config-volume\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.583437 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.583401 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbrk\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-kube-api-access-cwbrk\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.583629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.583478 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.583629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.583542 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-routing-overlay-volume\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.583629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.583579 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-plugins-config-volume\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.584167 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.584139 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-routing-overlay-volume\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.584368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.584346 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-plugins-config-volume\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.586025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.586009 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.636531 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.636465 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbrk\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-kube-api-access-cwbrk\") pod \"payload-processing-external-model-666cfbc7c7-mfd9w\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:17.694144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:17.694115 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:18.068424 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:18.068393 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w"] Nov 02 20:54:18.068851 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:54:18.068828 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a69ca54_da82_41be_86a4_e58826f3c2dd.slice/crio-f64ea11bbfeb309a97d928066c3db1a1b9de021f47ca14fd3521e33881fe8359 WatchSource:0}: Error finding container f64ea11bbfeb309a97d928066c3db1a1b9de021f47ca14fd3521e33881fe8359: Status 404 returned error can't find the container with id f64ea11bbfeb309a97d928066c3db1a1b9de021f47ca14fd3521e33881fe8359 Nov 02 20:54:19.043401 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:19.043358 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" event={"ID":"1a69ca54-da82-41be-86a4-e58826f3c2dd","Type":"ContainerStarted","Data":"fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7"} Nov 02 20:54:19.043938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:19.043665 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:19.043938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:19.043733 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" event={"ID":"1a69ca54-da82-41be-86a4-e58826f3c2dd","Type":"ContainerStarted","Data":"f64ea11bbfeb309a97d928066c3db1a1b9de021f47ca14fd3521e33881fe8359"} Nov 02 20:54:19.213767 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:19.213733 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:19.213767 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:19.213765 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:19.214054 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:19.213837 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials podName:1a69ca54-da82-41be-86a4-e58826f3c2dd nodeName:}" failed. No retries permitted until 2026-11-02 20:54:19.713817179 +0000 UTC m=+3958.230589758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials") pod "payload-processing-external-model-666cfbc7c7-mfd9w" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd") : secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:19.723040 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:19.723010 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:19.723040 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:19.723036 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:19.723259 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:19.723093 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials podName:1a69ca54-da82-41be-86a4-e58826f3c2dd nodeName:}" failed. No retries permitted until 2026-11-02 20:54:20.723078729 +0000 UTC m=+3959.239851305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials") pod "payload-processing-external-model-666cfbc7c7-mfd9w" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd") : secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:20.745306 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:20.744698 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:20.745306 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:20.744776 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:20.745306 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:20.744843 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials podName:1a69ca54-da82-41be-86a4-e58826f3c2dd nodeName:}" failed. No retries permitted until 2026-11-02 20:54:22.744823669 +0000 UTC m=+3961.261596236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials") pod "payload-processing-external-model-666cfbc7c7-mfd9w" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd") : secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:22.189319 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.189245 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" podStartSLOduration=5.189228411 podStartE2EDuration="5.189228411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:19.150704299 +0000 UTC m=+3957.667476889" watchObservedRunningTime="2026-11-02 20:54:22.189228411 +0000 UTC m=+3960.706000995" Nov 02 20:54:22.192263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.192234 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w"] Nov 02 20:54:22.232406 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.232377 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r"] Nov 02 20:54:22.236701 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.236682 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.330150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.326591 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r"] Nov 02 20:54:22.372140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.350457 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-689f7b1680fb\"" Nov 02 20:54:22.372429 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.371869 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-routing-overlay-volume\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.373338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.373314 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-provider-credentials\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.373504 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.373485 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86ddh\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-kube-api-access-86ddh\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.373602 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.373587 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-plugins-config-volume\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.475254 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.475167 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-routing-overlay-volume\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.475438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.475314 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-provider-credentials\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.475438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.475376 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86ddh\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-kube-api-access-86ddh\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.475438 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.475405 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-plugins-config-volume\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.475928 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.475900 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-routing-overlay-volume\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.476168 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.476145 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-plugins-config-volume\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.477883 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.477859 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-provider-credentials\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.528270 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.528236 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86ddh\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-kube-api-access-86ddh\") pod \"payload-processing-external-model-5dd9ffcf-rsm6r\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.556326 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.555875 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:22.706810 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:22.706600 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:54:22.779465 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:22.779432 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:22.779465 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:22.779465 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:22.779695 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:22.779542 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials podName:1a69ca54-da82-41be-86a4-e58826f3c2dd nodeName:}" failed. No retries permitted until 2026-11-02 20:54:26.779520053 +0000 UTC m=+3965.296292615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials") pod "payload-processing-external-model-666cfbc7c7-mfd9w" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd") : secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:22.958396 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:22.958369 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r"] Nov 02 20:54:22.959739 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:54:22.959712 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e0908_d3d9_400a_866d_1997a6ade2d0.slice/crio-820f253eec7ebe691e3134649d2eb624ac6b65f4a847de15d09768134efb3b3f WatchSource:0}: Error finding container 820f253eec7ebe691e3134649d2eb624ac6b65f4a847de15d09768134efb3b3f: Status 404 returned error can't find the container with id 820f253eec7ebe691e3134649d2eb624ac6b65f4a847de15d09768134efb3b3f Nov 02 20:54:23.072371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:23.072271 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" event={"ID":"c10e0908-d3d9-400a-866d-1997a6ade2d0","Type":"ContainerStarted","Data":"f96d4b5744b8c900c1d2062a0e0afa9ce8142e9d7349fea5c705c79776e0c79f"} Nov 02 20:54:23.072371 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:23.072357 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:23.072589 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:23.072383 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" event={"ID":"c10e0908-d3d9-400a-866d-1997a6ade2d0","Type":"ContainerStarted","Data":"820f253eec7ebe691e3134649d2eb624ac6b65f4a847de15d09768134efb3b3f"} Nov 02 20:54:23.181088 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:23.181030 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" podStartSLOduration=1.181016643 podStartE2EDuration="1.181016643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:23.17903141 +0000 UTC m=+3961.695803995" watchObservedRunningTime="2026-11-02 20:54:23.181016643 +0000 UTC m=+3961.697789227" Nov 02 20:54:25.488241 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:25.488204 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc"] Nov 02 20:54:25.502501 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:25.502462 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0a8f62-rsxbc"] Nov 02 20:54:25.637893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:25.637834 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def9a282-bf2a-4dc2-9687-03d42f0a4a41" path="/var/lib/kubelet/pods/def9a282-bf2a-4dc2-9687-03d42f0a4a41/volumes" Nov 02 20:54:26.837394 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:26.837355 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c233618dc3e5: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:26.837960 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:26.837930 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w: secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:26.838125 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:26.838030 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials podName:1a69ca54-da82-41be-86a4-e58826f3c2dd nodeName:}" failed. No retries permitted until 2026-11-02 20:54:34.838009075 +0000 UTC m=+3973.354781663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials") pod "payload-processing-external-model-666cfbc7c7-mfd9w" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd") : secret "agc-cred-c233618dc3e5" not found Nov 02 20:54:27.026783 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.026742 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r"] Nov 02 20:54:27.193185 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.193091 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" podUID="1a69ca54-da82-41be-86a4-e58826f3c2dd" containerName="payload-processing" containerID="cri-o://fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7" gracePeriod=25 Nov 02 20:54:27.467880 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.467851 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:27.547332 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.547297 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-plugins-config-volume\") pod \"1a69ca54-da82-41be-86a4-e58826f3c2dd\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " Nov 02 20:54:27.547544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.547344 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials\") pod \"1a69ca54-da82-41be-86a4-e58826f3c2dd\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " Nov 02 20:54:27.547544 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.547377 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-kube-api-access-cwbrk\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-kube-api-access-cwbrk\") pod \"1a69ca54-da82-41be-86a4-e58826f3c2dd\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " Nov 02 20:54:27.547708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.547648 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-routing-overlay-volume\") pod \"1a69ca54-da82-41be-86a4-e58826f3c2dd\" (UID: \"1a69ca54-da82-41be-86a4-e58826f3c2dd\") " Nov 02 20:54:27.547835 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.547798 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-plugins-config-volume" pod "1a69ca54-da82-41be-86a4-e58826f3c2dd" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:27.548041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.548017 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-routing-overlay-volume" pod "1a69ca54-da82-41be-86a4-e58826f3c2dd" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:27.548263 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.548245 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:27.548346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.548270 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a69ca54-da82-41be-86a4-e58826f3c2dd-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:27.549762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.549737 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials" pod "1a69ca54-da82-41be-86a4-e58826f3c2dd" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:27.549876 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.549804 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-kube-api-access-cwbrk" pod "1a69ca54-da82-41be-86a4-e58826f3c2dd" (UID: "1a69ca54-da82-41be-86a4-e58826f3c2dd"). InnerVolumeSpecName "kube-api-access-cwbrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:27.650234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.650180 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:27.650234 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.650210 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbrk\" (UniqueName: \"kubernetes.io/projected/1a69ca54-da82-41be-86a4-e58826f3c2dd-kube-api-access-cwbrk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:27.892092 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.892052 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp"] Nov 02 20:54:27.892756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.892741 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1a69ca54-da82-41be-86a4-e58826f3c2dd" containerName="payload-processing" Nov 02 20:54:27.892804 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.892757 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a69ca54-da82-41be-86a4-e58826f3c2dd" containerName="payload-processing" Nov 02 20:54:27.892863 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.892853 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a69ca54-da82-41be-86a4-e58826f3c2dd" containerName="payload-processing" Nov 02 20:54:27.896636 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.896609 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.954666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.954624 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378a9e33-b189-43ec-a595-195a873d45e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.954846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.954687 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf66s\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-kube-api-access-lf66s\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.954846 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.954793 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.954998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.954873 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.954998 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.954925 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378a9e33-b189-43ec-a595-195a873d45e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.955112 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.955089 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.955175 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.955142 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.955250 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.955227 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.955328 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.955314 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:27.997138 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:27.997102 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp"] Nov 02 20:54:28.056035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056000 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056050 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056144 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056172 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056231 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378a9e33-b189-43ec-a595-195a873d45e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056262 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf66s\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-kube-api-access-lf66s\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056370 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056359 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056424 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056471 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378a9e33-b189-43ec-a595-195a873d45e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056695 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056614 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056902 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.056932 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056922 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.057093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.056967 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.057419 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.057397 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378a9e33-b189-43ec-a595-195a873d45e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.059043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.059018 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.059249 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.059229 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378a9e33-b189-43ec-a595-195a873d45e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.107823 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.107791 5800 generic.go:356] "Generic (PLEG): container finished" podID="1a69ca54-da82-41be-86a4-e58826f3c2dd" containerID="fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7" exitCode=0 Nov 02 20:54:28.107978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.107861 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" Nov 02 20:54:28.107978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.107880 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" event={"ID":"1a69ca54-da82-41be-86a4-e58826f3c2dd","Type":"ContainerDied","Data":"fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7"} Nov 02 20:54:28.107978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.107960 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w" event={"ID":"1a69ca54-da82-41be-86a4-e58826f3c2dd","Type":"ContainerDied","Data":"f64ea11bbfeb309a97d928066c3db1a1b9de021f47ca14fd3521e33881fe8359"} Nov 02 20:54:28.107978 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.107973 5800 scope.go:122] "RemoveContainer" containerID="fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7" Nov 02 20:54:28.111041 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.111013 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.111192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.111083 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf66s\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-kube-api-access-lf66s\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-qt5rp\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.118008 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.117991 5800 scope.go:122] "RemoveContainer" containerID="fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7" Nov 02 20:54:28.118310 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:28.118292 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7\": container with ID starting with fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7 not found: ID does not exist" containerID="fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7" Nov 02 20:54:28.118357 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.118318 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7"} err="failed to get container status \"fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7\": rpc error: code = NotFound desc = could not find container \"fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7\": container with ID starting with fe53a397d6456acf26aac3a74e991fbe0a69440c22b3abbb1cd05634292160e7 not found: ID does not exist" Nov 02 20:54:28.210743 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.210664 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:28.262381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.262343 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w"] Nov 02 20:54:28.269982 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.269953 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-666cfbc7c7-mfd9w"] Nov 02 20:54:28.632162 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.632129 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp"] Nov 02 20:54:28.634130 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:54:28.634104 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378a9e33_b189_43ec_a595_195a873d45e1.slice/crio-59cfd3ddde435c4321d4a0dc8c7e6bb8944d85da3ce03a33acc9e8ff0539f9f0 WatchSource:0}: Error finding container 59cfd3ddde435c4321d4a0dc8c7e6bb8944d85da3ce03a33acc9e8ff0539f9f0: Status 404 returned error can't find the container with id 59cfd3ddde435c4321d4a0dc8c7e6bb8944d85da3ce03a33acc9e8ff0539f9f0 Nov 02 20:54:28.636820 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.636784 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:54:28.636929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.636865 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:54:28.636929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:28.636913 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:54:29.119113 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.119074 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" event={"ID":"378a9e33-b189-43ec-a595-195a873d45e1","Type":"ContainerStarted","Data":"2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e"} Nov 02 20:54:29.119517 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.119141 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" event={"ID":"378a9e33-b189-43ec-a595-195a873d45e1","Type":"ContainerStarted","Data":"59cfd3ddde435c4321d4a0dc8c7e6bb8944d85da3ce03a33acc9e8ff0539f9f0"} Nov 02 20:54:29.211756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.211717 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:29.213479 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.213454 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:29.213614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.213504 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:29.228414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.228350 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podStartSLOduration=2.2283332160000002 podStartE2EDuration="2.228333216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:29.227474908 +0000 UTC m=+3967.744247517" watchObservedRunningTime="2026-11-02 20:54:29.228333216 +0000 UTC m=+3967.745105801" Nov 02 20:54:29.457362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.456460 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n"] Nov 02 20:54:29.465317 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.465259 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.495259 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.495223 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n"] Nov 02 20:54:29.514821 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.514774 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f3109b-openshift-default-dockercfg-gwndp\"" Nov 02 20:54:29.599305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.585916 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-credential-socket\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.599305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.585980 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56831529-6563-43be-ac88-3166ffb219d2-istiod-ca-cert\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.599305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.586013 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-data\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.602414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.601423 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-envoy\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.602414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.601513 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtjj\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-kube-api-access-vrtjj\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.602414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.601553 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-istio-token\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.602414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.601706 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56831529-6563-43be-ac88-3166ffb219d2-istio-podinfo\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.602414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.601776 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-certs\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.602414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.601826 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-socket\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.676546 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.676509 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a69ca54-da82-41be-86a4-e58826f3c2dd" path="/var/lib/kubelet/pods/1a69ca54-da82-41be-86a4-e58826f3c2dd/volumes" Nov 02 20:54:29.703360 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.703324 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-envoy\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.703543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.703401 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtjj\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-kube-api-access-vrtjj\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.703543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.703437 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-istio-token\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.703726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.703700 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56831529-6563-43be-ac88-3166ffb219d2-istio-podinfo\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.703829 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.703770 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-certs\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.703934 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.703881 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-socket\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704021 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.703997 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-credential-socket\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704110 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.704039 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56831529-6563-43be-ac88-3166ffb219d2-istiod-ca-cert\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704110 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.704073 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-data\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704268 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.704159 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-certs\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.704331 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-socket\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704673 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.704625 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-credential-socket\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704760 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.704721 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-data\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.704839 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.704820 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56831529-6563-43be-ac88-3166ffb219d2-istiod-ca-cert\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.705923 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.705903 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-envoy\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.706915 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.706894 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56831529-6563-43be-ac88-3166ffb219d2-istio-podinfo\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.771307 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.765993 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-istio-token\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:29.787002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:29.786963 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtjj\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-kube-api-access-vrtjj\") pod \"e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:30.080951 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:30.080856 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:30.212304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:30.212238 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:30.212773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:30.212310 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:30.557316 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:30.556626 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n"] Nov 02 20:54:30.560071 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:30.560032 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:54:30.560199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:30.560123 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:54:30.560199 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:30.560168 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:54:31.163150 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:31.163109 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" event={"ID":"56831529-6563-43be-ac88-3166ffb219d2","Type":"ContainerStarted","Data":"da46f1a09931a4ce1a62c0c8377aaeb916b00d46eb084e739b4d7e17774564e5"} Nov 02 20:54:31.163362 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:31.163173 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" event={"ID":"56831529-6563-43be-ac88-3166ffb219d2","Type":"ContainerStarted","Data":"63dee6818a746626eaa8c5a3e9e6438170a302c74764d6558f437da85d3a354c"} Nov 02 20:54:31.211709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:31.211666 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:31.211929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:31.211728 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:31.312206 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:31.312148 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podStartSLOduration=2.312130851 podStartE2EDuration="2.312130851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:54:31.271207423 +0000 UTC m=+3969.787980008" watchObservedRunningTime="2026-11-02 20:54:31.312130851 +0000 UTC m=+3969.828903438" Nov 02 20:54:32.030394 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.030346 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" podUID="c10e0908-d3d9-400a-866d-1997a6ade2d0" containerName="payload-processing" containerID="cri-o://f96d4b5744b8c900c1d2062a0e0afa9ce8142e9d7349fea5c705c79776e0c79f" gracePeriod=25 Nov 02 20:54:32.081392 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.081354 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:32.090172 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.090132 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:32.183990 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.183956 5800 generic.go:356] "Generic (PLEG): container finished" podID="c10e0908-d3d9-400a-866d-1997a6ade2d0" containerID="f96d4b5744b8c900c1d2062a0e0afa9ce8142e9d7349fea5c705c79776e0c79f" exitCode=0 Nov 02 20:54:32.184192 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.184032 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" event={"ID":"c10e0908-d3d9-400a-866d-1997a6ade2d0","Type":"ContainerDied","Data":"f96d4b5744b8c900c1d2062a0e0afa9ce8142e9d7349fea5c705c79776e0c79f"} Nov 02 20:54:32.184594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.184559 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:32.186140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.186115 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:54:32.212745 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.212029 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:32.212745 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.212094 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:32.809220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.809191 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:32.813909 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.813883 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-provider-credentials\") pod \"c10e0908-d3d9-400a-866d-1997a6ade2d0\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " Nov 02 20:54:32.814093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.813996 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-routing-overlay-volume\") pod \"c10e0908-d3d9-400a-866d-1997a6ade2d0\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " Nov 02 20:54:32.814093 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.814065 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-plugins-config-volume\") pod \"c10e0908-d3d9-400a-866d-1997a6ade2d0\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " Nov 02 20:54:32.814218 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.814197 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-kube-api-access-86ddh\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-kube-api-access-86ddh\") pod \"c10e0908-d3d9-400a-866d-1997a6ade2d0\" (UID: \"c10e0908-d3d9-400a-866d-1997a6ade2d0\") " Nov 02 20:54:32.814650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.814626 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-routing-overlay-volume" pod "c10e0908-d3d9-400a-866d-1997a6ade2d0" (UID: "c10e0908-d3d9-400a-866d-1997a6ade2d0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:32.815348 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.815325 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-plugins-config-volume" pod "c10e0908-d3d9-400a-866d-1997a6ade2d0" (UID: "c10e0908-d3d9-400a-866d-1997a6ade2d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:54:32.816812 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.816791 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-kube-api-access-86ddh" pod "c10e0908-d3d9-400a-866d-1997a6ade2d0" (UID: "c10e0908-d3d9-400a-866d-1997a6ade2d0"). InnerVolumeSpecName "kube-api-access-86ddh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:32.817352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.817335 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-provider-credentials" pod "c10e0908-d3d9-400a-866d-1997a6ade2d0" (UID: "c10e0908-d3d9-400a-866d-1997a6ade2d0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:54:32.915885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.915846 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:32.915885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.915880 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c10e0908-d3d9-400a-866d-1997a6ade2d0-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:32.915885 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.915890 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86ddh\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-kube-api-access-86ddh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:32.916128 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:32.915899 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c10e0908-d3d9-400a-866d-1997a6ade2d0-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:54:33.194650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.194558 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" Nov 02 20:54:33.194650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.194558 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r" event={"ID":"c10e0908-d3d9-400a-866d-1997a6ade2d0","Type":"ContainerDied","Data":"820f253eec7ebe691e3134649d2eb624ac6b65f4a847de15d09768134efb3b3f"} Nov 02 20:54:33.194650 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.194650 5800 scope.go:122] "RemoveContainer" containerID="f96d4b5744b8c900c1d2062a0e0afa9ce8142e9d7349fea5c705c79776e0c79f" Nov 02 20:54:33.211641 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.211614 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:33.211762 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.211662 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:33.379963 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.379926 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r"] Nov 02 20:54:33.387872 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.387841 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dd9ffcf-rsm6r"] Nov 02 20:54:33.657306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:33.653596 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10e0908-d3d9-400a-866d-1997a6ade2d0" path="/var/lib/kubelet/pods/c10e0908-d3d9-400a-866d-1997a6ade2d0/volumes" Nov 02 20:54:34.211245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:34.211213 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:34.211665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:34.211264 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:35.211816 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:35.211781 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:35.212236 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:35.211835 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:36.211428 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:36.211389 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 02 20:54:36.211657 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:36.211449 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 02 20:54:37.526032 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:37.525979 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:37.527207 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:37.527175 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:37.527942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:54:37.527919 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 20:54:52.696721 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:54:52.696678 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:55:08.158722 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.158676 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6"] Nov 02 20:55:08.159627 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.159608 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c10e0908-d3d9-400a-866d-1997a6ade2d0" containerName="payload-processing" Nov 02 20:55:08.159682 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.159630 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10e0908-d3d9-400a-866d-1997a6ade2d0" containerName="payload-processing" Nov 02 20:55:08.159798 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.159786 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10e0908-d3d9-400a-866d-1997a6ade2d0" containerName="payload-processing" Nov 02 20:55:08.164243 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.164225 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.215369 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.215339 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6"] Nov 02 20:55:08.269991 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.269907 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1dd5d5-dockercfg-ddkm8\"" Nov 02 20:55:08.271890 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.270921 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1dd5d5\"" Nov 02 20:55:08.277385 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.271763 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088dcad0-68df-4eb4-9b52-5e7662b160c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.299349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.290943 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdps\" (UniqueName: \"kubernetes.io/projected/088dcad0-68df-4eb4-9b52-5e7662b160c3-kube-api-access-frdps\") pod \"payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.320543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.309587 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv"] Nov 02 20:55:08.354335 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.354236 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.382979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.382945 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv"] Nov 02 20:55:08.421676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.421575 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088dcad0-68df-4eb4-9b52-5e7662b160c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.421676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.421668 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjv9\" (UniqueName: \"kubernetes.io/projected/0e3e88e8-c62a-4e11-a732-793d093ef827-kube-api-access-2fjv9\") pod \"payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.421944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.421723 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3e88e8-c62a-4e11-a732-793d093ef827-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.421944 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.421802 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdps\" (UniqueName: \"kubernetes.io/projected/088dcad0-68df-4eb4-9b52-5e7662b160c3-kube-api-access-frdps\") pod \"payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.422451 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.422427 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088dcad0-68df-4eb4-9b52-5e7662b160c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.477533 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.477493 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdps\" (UniqueName: \"kubernetes.io/projected/088dcad0-68df-4eb4-9b52-5e7662b160c3-kube-api-access-frdps\") pod \"payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.480301 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.480255 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:08.522675 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.522634 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjv9\" (UniqueName: \"kubernetes.io/projected/0e3e88e8-c62a-4e11-a732-793d093ef827-kube-api-access-2fjv9\") pod \"payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.522870 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.522719 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3e88e8-c62a-4e11-a732-793d093ef827-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.523453 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.523433 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3e88e8-c62a-4e11-a732-793d093ef827-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.575551 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.575425 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjv9\" (UniqueName: \"kubernetes.io/projected/0e3e88e8-c62a-4e11-a732-793d093ef827-kube-api-access-2fjv9\") pod \"payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.688160 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.688073 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:08.883676 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:08.883649 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6"] Nov 02 20:55:08.885159 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:55:08.885134 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088dcad0_68df_4eb4_9b52_5e7662b160c3.slice/crio-f6ac9a3450c1d9e1d2eb3a1f9dc577d64deb2d0f75be321355cea342ead0c059 WatchSource:0}: Error finding container f6ac9a3450c1d9e1d2eb3a1f9dc577d64deb2d0f75be321355cea342ead0c059: Status 404 returned error can't find the container with id f6ac9a3450c1d9e1d2eb3a1f9dc577d64deb2d0f75be321355cea342ead0c059 Nov 02 20:55:09.065010 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.064984 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv"] Nov 02 20:55:09.066549 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:55:09.066517 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3e88e8_c62a_4e11_a732_793d093ef827.slice/crio-af007d5e81adab6bb01cc8b1eb4272ccc660786fcef3f2d710597a76b2ea8fd2 WatchSource:0}: Error finding container af007d5e81adab6bb01cc8b1eb4272ccc660786fcef3f2d710597a76b2ea8fd2: Status 404 returned error can't find the container with id af007d5e81adab6bb01cc8b1eb4272ccc660786fcef3f2d710597a76b2ea8fd2 Nov 02 20:55:09.507613 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.507575 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" event={"ID":"0e3e88e8-c62a-4e11-a732-793d093ef827","Type":"ContainerStarted","Data":"1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc"} Nov 02 20:55:09.508114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.507672 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" event={"ID":"0e3e88e8-c62a-4e11-a732-793d093ef827","Type":"ContainerStarted","Data":"af007d5e81adab6bb01cc8b1eb4272ccc660786fcef3f2d710597a76b2ea8fd2"} Nov 02 20:55:09.508114 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.507823 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:09.509460 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.509435 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" event={"ID":"088dcad0-68df-4eb4-9b52-5e7662b160c3","Type":"ContainerStarted","Data":"5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe"} Nov 02 20:55:09.509610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.509474 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" event={"ID":"088dcad0-68df-4eb4-9b52-5e7662b160c3","Type":"ContainerStarted","Data":"f6ac9a3450c1d9e1d2eb3a1f9dc577d64deb2d0f75be321355cea342ead0c059"} Nov 02 20:55:09.509610 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.509516 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:09.613025 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.612961 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" podStartSLOduration=1.612942276 podStartE2EDuration="1.612942276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:09.6124197 +0000 UTC m=+4008.129192285" watchObservedRunningTime="2026-11-02 20:55:09.612942276 +0000 UTC m=+4008.129714862" Nov 02 20:55:09.719404 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:09.719344 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" podStartSLOduration=1.719327366 podStartE2EDuration="1.719327366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:09.71739057 +0000 UTC m=+4008.234163157" watchObservedRunningTime="2026-11-02 20:55:09.719327366 +0000 UTC m=+4008.236099952" Nov 02 20:55:13.223943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.223908 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42"] Nov 02 20:55:13.228647 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.228615 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.270342 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.270311 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx"] Nov 02 20:55:13.275140 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.275116 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.298854 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.298823 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6lh\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-kube-api-access-hd6lh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.298886 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299001 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.298982 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.299027 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299103 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.299052 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299183 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.299158 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299256 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.299239 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299350 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.299332 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.299434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.299419 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.356077 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.356040 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx"] Nov 02 20:55:13.400244 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400213 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6lh\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-kube-api-access-hd6lh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.400485 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400293 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.400637 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400610 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.400687 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400622 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.400728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400697 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.400728 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400719 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.400844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400754 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.400844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400780 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.400844 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.400796 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.401052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401003 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.401174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401114 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6ls\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-kube-api-access-vg6ls\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.401174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401123 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.401272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401183 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.401272 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401215 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.401408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401340 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.401408 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401397 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.401514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401426 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.401514 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401505 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.401700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401515 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.401700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401545 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.401700 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401597 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.401972 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.401949 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.402136 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.402113 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.404252 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.404230 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.404469 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.404451 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.416666 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.416453 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42"] Nov 02 20:55:13.453824 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.453785 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.453984 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.453873 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6lh\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-kube-api-access-hd6lh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-6bv42\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.502742 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.502696 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.502942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.502858 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.502942 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.502904 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.502941 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6ls\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-kube-api-access-vg6ls\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.502965 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503083 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503050 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503127 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503180 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503262 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503186 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503274 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503309 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503447 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503435 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503625 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503496 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.503973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.503951 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.505439 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.505418 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.505773 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.505752 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.549245 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.549208 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:13.556181 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.556149 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6ls\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-kube-api-access-vg6ls\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.556388 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.556368 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-wfqpx\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.589929 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.589897 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:13.983616 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.983581 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42"] Nov 02 20:55:13.984034 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:55:13.984006 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7a1b5f_0d72_480b_b514_f8cee2f619c8.slice/crio-d4410f74a6b9e43ec64c0d108ac35135796d8e9dfeb9f541e0e1f759f507ad46 WatchSource:0}: Error finding container d4410f74a6b9e43ec64c0d108ac35135796d8e9dfeb9f541e0e1f759f507ad46: Status 404 returned error can't find the container with id d4410f74a6b9e43ec64c0d108ac35135796d8e9dfeb9f541e0e1f759f507ad46 Nov 02 20:55:13.987044 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.987007 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:55:13.987139 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.987091 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:55:13.987190 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:13.987134 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:55:14.031530 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.031473 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx"] Nov 02 20:55:14.032783 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:55:14.032756 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0704c06f_bf67_4bf7_9a4b_2300e00f14e7.slice/crio-8038fa68869002f0d9e410d489b5b20ee4590f9a496e4cf72b588d13df42ebcb WatchSource:0}: Error finding container 8038fa68869002f0d9e410d489b5b20ee4590f9a496e4cf72b588d13df42ebcb: Status 404 returned error can't find the container with id 8038fa68869002f0d9e410d489b5b20ee4590f9a496e4cf72b588d13df42ebcb Nov 02 20:55:14.035689 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.035651 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:55:14.035805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.035738 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:55:14.035805 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.035787 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:55:14.553718 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.553676 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" event={"ID":"0704c06f-bf67-4bf7-9a4b-2300e00f14e7","Type":"ContainerStarted","Data":"5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce"} Nov 02 20:55:14.554220 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.553735 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" event={"ID":"0704c06f-bf67-4bf7-9a4b-2300e00f14e7","Type":"ContainerStarted","Data":"8038fa68869002f0d9e410d489b5b20ee4590f9a496e4cf72b588d13df42ebcb"} Nov 02 20:55:14.557346 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.557316 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" event={"ID":"4c7a1b5f-0d72-480b-b514-f8cee2f619c8","Type":"ContainerStarted","Data":"2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e"} Nov 02 20:55:14.557588 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.557569 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" event={"ID":"4c7a1b5f-0d72-480b-b514-f8cee2f619c8","Type":"ContainerStarted","Data":"d4410f74a6b9e43ec64c0d108ac35135796d8e9dfeb9f541e0e1f759f507ad46"} Nov 02 20:55:14.590527 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.590491 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:14.664403 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.664347 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podStartSLOduration=1.664332243 podStartE2EDuration="1.664332243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:14.661786437 +0000 UTC m=+4013.178559024" watchObservedRunningTime="2026-11-02 20:55:14.664332243 +0000 UTC m=+4013.181104827" Nov 02 20:55:14.771203 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:14.771141 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podStartSLOduration=1.77112582 podStartE2EDuration="1.77112582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:55:14.769913648 +0000 UTC m=+4013.286686226" watchObservedRunningTime="2026-11-02 20:55:14.77112582 +0000 UTC m=+4013.287898404" Nov 02 20:55:15.554306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:15.553635 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:15.568311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:15.563421 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:15.568311 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:15.563483 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:15.596219 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:15.596185 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 02 20:55:15.596525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:15.596499 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 02 20:55:16.551304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:16.550697 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:16.551304 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:16.550765 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:16.594320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:16.594187 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 02 20:55:16.594320 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:16.594245 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 02 20:55:17.550715 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:17.550677 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:17.551054 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:17.551006 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:17.592223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:17.592188 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 02 20:55:17.592525 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:17.592497 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 02 20:55:18.555303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:18.550697 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:18.555303 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:18.550776 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:18.591330 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:18.591258 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 02 20:55:18.591553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:18.591354 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 02 20:55:19.550306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:19.549673 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:19.550306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:19.549746 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:19.593482 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:19.593443 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 02 20:55:19.594106 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:19.594062 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 02 20:55:20.520926 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:20.520896 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:55:20.521267 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:20.521245 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:55:20.553861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:20.553826 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:20.554156 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:20.554122 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:20.595305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:20.594913 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 02 20:55:20.595305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:20.594998 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 02 20:55:21.549817 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:21.549779 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:21.550030 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:21.549848 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:21.590511 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:21.590466 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 02 20:55:21.590712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:21.590530 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 02 20:55:22.550612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:22.550571 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 02 20:55:22.551219 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:22.551183 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 02 20:55:22.718489 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:55:22.718441 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:55:23.544065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:23.542618 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:23.544065 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:23.543912 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:23.545081 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:23.545060 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 20:55:23.723583 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:23.723549 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:23.724693 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:23.724669 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:23.726127 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:55:23.726096 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 20:55:52.697213 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:55:52.697178 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:56:04.007387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:04.002997 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:56:22.697171 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:56:22.697125 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:56:33.225435 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:33.225398 5800 patch_prober.go:29] interesting pod/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:56:33.225935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:33.225464 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:56:35.861553 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:35.861512 5800 generic.go:356] "Generic (PLEG): container finished" podID="56831529-6563-43be-ac88-3166ffb219d2" containerID="da46f1a09931a4ce1a62c0c8377aaeb916b00d46eb084e739b4d7e17774564e5" exitCode=0 Nov 02 20:56:35.862133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:35.861573 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" event={"ID":"56831529-6563-43be-ac88-3166ffb219d2","Type":"ContainerDied","Data":"da46f1a09931a4ce1a62c0c8377aaeb916b00d46eb084e739b4d7e17774564e5"} Nov 02 20:56:35.862133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:35.861967 5800 scope.go:122] "RemoveContainer" containerID="da46f1a09931a4ce1a62c0c8377aaeb916b00d46eb084e739b4d7e17774564e5" Nov 02 20:56:35.863698 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:35.863666 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:56:35.863801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:35.863745 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:56:35.863801 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:35.863785 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:56:36.872665 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:36.872625 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" event={"ID":"56831529-6563-43be-ac88-3166ffb219d2","Type":"ContainerStarted","Data":"5914595b7c768bf2560e3a9cf4e0e35a3b5ce1675ad7f8656c0fddc88bd4c91b"} Nov 02 20:56:37.081586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:37.081544 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:56:37.081800 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:37.081768 5800 patch_prober.go:29] interesting pod/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 02 20:56:37.081861 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:37.081834 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 02 20:56:38.082051 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:38.082015 5800 patch_prober.go:29] interesting pod/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 02 20:56:38.082468 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:38.082085 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 02 20:56:39.082031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:39.081991 5800 patch_prober.go:29] interesting pod/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 02 20:56:39.082218 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:39.082054 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 02 20:56:40.082184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:40.082145 5800 patch_prober.go:29] interesting pod/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 02 20:56:40.082416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:40.082213 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 02 20:56:41.081939 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:41.081899 5800 patch_prober.go:29] interesting pod/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 02 20:56:41.082149 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:41.081954 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 02 20:56:42.082064 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:42.082018 5800 patch_prober.go:29] interesting pod/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 02 20:56:42.082537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:42.082098 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 02 20:56:43.086629 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:43.086600 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:56:43.087223 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:43.087195 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:56:43.087799 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:56:43.087780 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 20:56:52.699805 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:56:52.699773 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:57:20.899712 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:20.899675 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5"] Nov 02 20:57:20.904720 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:20.904697 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:20.974985 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:20.974945 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e19d40-openshift-default-dockercfg-qdk7b\"" Nov 02 20:57:21.000914 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.000875 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-podinfo\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001053 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-token\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001096 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-envoy\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001120 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096d67b-03cb-45da-b019-d36fe8dc3f97-istiod-ca-cert\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001150 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fls5\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-kube-api-access-7fls5\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001234 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-credential-socket\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001338 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-data\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001380 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-certs\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.001472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.001419 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-socket\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.025266 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.025236 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5"] Nov 02 20:57:21.102445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102407 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-certs\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.102445 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102448 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-socket\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.102754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102532 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-podinfo\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.102754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102601 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-token\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.102754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102629 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-envoy\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.102754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102653 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096d67b-03cb-45da-b019-d36fe8dc3f97-istiod-ca-cert\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.102754 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102683 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fls5\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-kube-api-access-7fls5\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.103052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102756 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-credential-socket\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.103052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102821 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-data\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.103052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.102970 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-socket\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.103224 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.103153 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-data\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.103470 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.103407 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-credential-socket\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.103631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.103502 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-certs\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.103631 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.103509 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096d67b-03cb-45da-b019-d36fe8dc3f97-istiod-ca-cert\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.105002 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.104978 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-envoy\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.105264 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.105246 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-podinfo\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.202891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.201912 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-token\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.204865 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.204796 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fls5\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-kube-api-access-7fls5\") pod \"e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.219651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.219614 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:21.751063 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:57:21.751028 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode096d67b_03cb_45da_b019_d36fe8dc3f97.slice/crio-c416d663c23541e3e9154d2388ebc4a06db72f1785f4ae9516c09296183e8cb3 WatchSource:0}: Error finding container c416d663c23541e3e9154d2388ebc4a06db72f1785f4ae9516c09296183e8cb3: Status 404 returned error can't find the container with id c416d663c23541e3e9154d2388ebc4a06db72f1785f4ae9516c09296183e8cb3 Nov 02 20:57:21.751873 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.751843 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5"] Nov 02 20:57:21.754016 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.753979 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:57:21.754104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.754048 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:57:21.754104 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:21.754084 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:57:22.334131 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:22.334092 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" event={"ID":"e096d67b-03cb-45da-b019-d36fe8dc3f97","Type":"ContainerStarted","Data":"baa17535f2aa6e65bf7de744efbe3b8c77f876d0c4349087b50ab2eed5076dcd"} Nov 02 20:57:22.334679 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:22.334155 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" event={"ID":"e096d67b-03cb-45da-b019-d36fe8dc3f97","Type":"ContainerStarted","Data":"c416d663c23541e3e9154d2388ebc4a06db72f1785f4ae9516c09296183e8cb3"} Nov 02 20:57:22.445242 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:22.445181 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" podStartSLOduration=2.445168623 podStartE2EDuration="2.445168623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:22.441810003 +0000 UTC m=+4140.958582589" watchObservedRunningTime="2026-11-02 20:57:22.445168623 +0000 UTC m=+4140.961941208" Nov 02 20:57:22.697674 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:57:22.697590 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:57:23.220971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:23.220937 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:23.228472 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:23.228435 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:23.343155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:23.343126 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:23.344211 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:23.344193 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:57:29.506031 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.505998 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv"] Nov 02 20:57:29.510560 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.510527 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.574306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.566520 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv"] Nov 02 20:57:29.591792 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.591735 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gct\" (UniqueName: \"kubernetes.io/projected/f260f4d0-4948-491a-b22e-772c461425b4-kube-api-access-x4gct\") pod \"payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.592225 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.592190 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f260f4d0-4948-491a-b22e-772c461425b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.606606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.606548 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e19d40-dockercfg-x6sxs\"" Nov 02 20:57:29.606606 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.606561 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e19d40\"" Nov 02 20:57:29.696308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.694234 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gct\" (UniqueName: \"kubernetes.io/projected/f260f4d0-4948-491a-b22e-772c461425b4-kube-api-access-x4gct\") pod \"payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.696308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.694427 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f260f4d0-4948-491a-b22e-772c461425b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.713979 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.713881 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f260f4d0-4948-491a-b22e-772c461425b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.718573 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.718529 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72"] Nov 02 20:57:29.730174 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.730142 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:29.759614 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.759534 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gct\" (UniqueName: \"kubernetes.io/projected/f260f4d0-4948-491a-b22e-772c461425b4-kube-api-access-x4gct\") pod \"payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.760510 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.760464 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72"] Nov 02 20:57:29.796786 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.796747 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-kube-api-access-bdjjw\") pod \"payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:29.796987 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.796935 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:29.826269 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.826233 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:29.898906 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.898869 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:29.899133 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.899105 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-kube-api-access-bdjjw\") pod \"payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:29.900135 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.900112 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:29.956050 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:29.956012 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-kube-api-access-bdjjw\") pod \"payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:30.095052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:30.094962 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:30.212654 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:57:30.212619 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf260f4d0_4948_491a_b22e_772c461425b4.slice/crio-5ea8e83475565173aa40e9eb64c41a821e073fb67b76eb0a1c963f3e1ee5c184 WatchSource:0}: Error finding container 5ea8e83475565173aa40e9eb64c41a821e073fb67b76eb0a1c963f3e1ee5c184: Status 404 returned error can't find the container with id 5ea8e83475565173aa40e9eb64c41a821e073fb67b76eb0a1c963f3e1ee5c184 Nov 02 20:57:30.214595 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:30.214563 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv"] Nov 02 20:57:30.404732 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:30.404692 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" event={"ID":"f260f4d0-4948-491a-b22e-772c461425b4","Type":"ContainerStarted","Data":"e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250"} Nov 02 20:57:30.404908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:30.404745 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" event={"ID":"f260f4d0-4948-491a-b22e-772c461425b4","Type":"ContainerStarted","Data":"5ea8e83475565173aa40e9eb64c41a821e073fb67b76eb0a1c963f3e1ee5c184"} Nov 02 20:57:30.404908 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:30.404788 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:30.483651 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:30.483620 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72"] Nov 02 20:57:30.483988 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:57:30.483964 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a927c6_b344_4c65_8ce6_39a21a3f4fa4.slice/crio-47d47e5e97c8c1b5fa7a02fe889d2cd6836223918addf48be6012a52a58ba377 WatchSource:0}: Error finding container 47d47e5e97c8c1b5fa7a02fe889d2cd6836223918addf48be6012a52a58ba377: Status 404 returned error can't find the container with id 47d47e5e97c8c1b5fa7a02fe889d2cd6836223918addf48be6012a52a58ba377 Nov 02 20:57:30.558957 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:30.558899 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" podStartSLOduration=1.558879439 podStartE2EDuration="1.558879439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:30.50990142 +0000 UTC m=+4149.026674007" watchObservedRunningTime="2026-11-02 20:57:30.558879439 +0000 UTC m=+4149.075652028" Nov 02 20:57:31.416069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:31.416029 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" event={"ID":"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4","Type":"ContainerStarted","Data":"c1a2bcf1f303eb44177000cfce731cf2d70da5399e87b879b11cb708c78ff03c"} Nov 02 20:57:31.416265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:31.416091 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" event={"ID":"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4","Type":"ContainerStarted","Data":"47d47e5e97c8c1b5fa7a02fe889d2cd6836223918addf48be6012a52a58ba377"} Nov 02 20:57:31.416265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:31.416146 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:40.638943 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.638877 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" podStartSLOduration=11.638857980000001 podStartE2EDuration="11.63885798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:31.571571067 +0000 UTC m=+4150.088343652" watchObservedRunningTime="2026-11-02 20:57:40.63885798 +0000 UTC m=+4159.155630567" Nov 02 20:57:40.639888 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.639865 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h"] Nov 02 20:57:40.645416 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.645393 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:40.699351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.692075 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b299a-2dde-4d3b-bd34-0a527e4af184-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:40.699351 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.692378 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgjxx\" (UniqueName: \"kubernetes.io/projected/7a2b299a-2dde-4d3b-bd34-0a527e4af184-kube-api-access-sgjxx\") pod \"payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:40.745306 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.726863 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h"] Nov 02 20:57:40.753305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.745619 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-801923-dockercfg-fn5dj\"" Nov 02 20:57:40.753305 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.745709 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-801923\"" Nov 02 20:57:40.804308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.799516 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b299a-2dde-4d3b-bd34-0a527e4af184-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:40.804308 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.799739 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgjxx\" (UniqueName: \"kubernetes.io/projected/7a2b299a-2dde-4d3b-bd34-0a527e4af184-kube-api-access-sgjxx\") pod \"payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:40.837222 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.837184 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b299a-2dde-4d3b-bd34-0a527e4af184-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:40.879462 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.879428 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgjxx\" (UniqueName: \"kubernetes.io/projected/7a2b299a-2dde-4d3b-bd34-0a527e4af184-kube-api-access-sgjxx\") pod \"payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:40.911042 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.910960 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg"] Nov 02 20:57:40.918579 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.918549 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:40.925052 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.925020 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg"] Nov 02 20:57:40.965309 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:40.965263 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:41.108389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.106402 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2347be97-13a0-4971-9f67-886f644c77a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:41.108389 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.106534 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtn4\" (UniqueName: \"kubernetes.io/projected/2347be97-13a0-4971-9f67-886f644c77a4-kube-api-access-7gtn4\") pod \"payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:41.207724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.207638 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2347be97-13a0-4971-9f67-886f644c77a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:41.207724 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.207703 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtn4\" (UniqueName: \"kubernetes.io/projected/2347be97-13a0-4971-9f67-886f644c77a4-kube-api-access-7gtn4\") pod \"payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:41.208417 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.208395 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2347be97-13a0-4971-9f67-886f644c77a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:41.261056 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.261012 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtn4\" (UniqueName: \"kubernetes.io/projected/2347be97-13a0-4971-9f67-886f644c77a4-kube-api-access-7gtn4\") pod \"payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:41.338398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.338366 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h"] Nov 02 20:57:41.339543 ip-10-0-134-13 kubenswrapper[5800]: W1102 20:57:41.339514 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2b299a_2dde_4d3b_bd34_0a527e4af184.slice/crio-680cab9dd24dbc3e5116b863a20d0a1910399e0b7a9d88cf0d14933b75917227 WatchSource:0}: Error finding container 680cab9dd24dbc3e5116b863a20d0a1910399e0b7a9d88cf0d14933b75917227: Status 404 returned error can't find the container with id 680cab9dd24dbc3e5116b863a20d0a1910399e0b7a9d88cf0d14933b75917227 Nov 02 20:57:41.418612 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.418590 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 20:57:41.497069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.496984 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" event={"ID":"7a2b299a-2dde-4d3b-bd34-0a527e4af184","Type":"ContainerStarted","Data":"a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f"} Nov 02 20:57:41.497069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.497045 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:41.497069 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.497063 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" event={"ID":"7a2b299a-2dde-4d3b-bd34-0a527e4af184","Type":"ContainerStarted","Data":"680cab9dd24dbc3e5116b863a20d0a1910399e0b7a9d88cf0d14933b75917227"} Nov 02 20:57:41.547144 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.547112 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:41.652910 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.648297 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" podStartSLOduration=1.648257981 podStartE2EDuration="1.648257981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:41.646031428 +0000 UTC m=+4160.162804014" watchObservedRunningTime="2026-11-02 20:57:41.648257981 +0000 UTC m=+4160.165030567" Nov 02 20:57:41.948434 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:41.948266 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg"] Nov 02 20:57:42.506814 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:42.506772 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" event={"ID":"2347be97-13a0-4971-9f67-886f644c77a4","Type":"ContainerStarted","Data":"891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39"} Nov 02 20:57:42.507035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:42.506834 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" event={"ID":"2347be97-13a0-4971-9f67-886f644c77a4","Type":"ContainerStarted","Data":"2613e5c30425731857a6f43a43f0261782bc3e8d1df6be85932fa629c5d64da1"} Nov 02 20:57:42.507035 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:42.506868 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:57:42.614397 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:42.614338 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" podStartSLOduration=2.6143216369999998 podStartE2EDuration="2.614321637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:57:42.612263024 +0000 UTC m=+4161.129035615" watchObservedRunningTime="2026-11-02 20:57:42.614321637 +0000 UTC m=+4161.131094225" Nov 02 20:57:43.435048 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:43.435015 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 20:57:52.510473 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:52.510441 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 20:57:52.698035 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:57:52.698004 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:57:53.519516 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:57:53.519487 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 20:58:09.829349 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:09.829304 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv"] Nov 02 20:58:09.956983 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:09.956944 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6"] Nov 02 20:58:14.829726 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:14.829685 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" podUID="0e3e88e8-c62a-4e11-a732-793d093ef827" containerName="payload-processing" containerID="cri-o://1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc" gracePeriod=25 Nov 02 20:58:14.957455 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:14.957403 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" podUID="088dcad0-68df-4eb4-9b52-5e7662b160c3" containerName="payload-pre-processing" containerID="cri-o://5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe" gracePeriod=25 Nov 02 20:58:15.133543 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.133515 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:58:15.238338 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.238307 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3e88e8-c62a-4e11-a732-793d093ef827-kube-api-access-2fjv9\" (UniqueName: \"kubernetes.io/projected/0e3e88e8-c62a-4e11-a732-793d093ef827-kube-api-access-2fjv9\") pod \"0e3e88e8-c62a-4e11-a732-793d093ef827\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " Nov 02 20:58:15.238537 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.238518 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3e88e8-c62a-4e11-a732-793d093ef827-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3e88e8-c62a-4e11-a732-793d093ef827-plugins-config-volume\") pod \"0e3e88e8-c62a-4e11-a732-793d093ef827\" (UID: \"0e3e88e8-c62a-4e11-a732-793d093ef827\") " Nov 02 20:58:15.238961 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.238918 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3e88e8-c62a-4e11-a732-793d093ef827-plugins-config-volume" pod "0e3e88e8-c62a-4e11-a732-793d093ef827" (UID: "0e3e88e8-c62a-4e11-a732-793d093ef827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:58:15.239202 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.239165 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3e88e8-c62a-4e11-a732-793d093ef827-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:58:15.241662 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.241616 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3e88e8-c62a-4e11-a732-793d093ef827-kube-api-access-2fjv9" pod "0e3e88e8-c62a-4e11-a732-793d093ef827" (UID: "0e3e88e8-c62a-4e11-a732-793d093ef827"). InnerVolumeSpecName "kube-api-access-2fjv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:58:15.246387 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.246366 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:58:15.340398 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.340366 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088dcad0-68df-4eb4-9b52-5e7662b160c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088dcad0-68df-4eb4-9b52-5e7662b160c3-plugins-config-volume\") pod \"088dcad0-68df-4eb4-9b52-5e7662b160c3\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " Nov 02 20:58:15.340594 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.340488 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088dcad0-68df-4eb4-9b52-5e7662b160c3-kube-api-access-frdps\" (UniqueName: \"kubernetes.io/projected/088dcad0-68df-4eb4-9b52-5e7662b160c3-kube-api-access-frdps\") pod \"088dcad0-68df-4eb4-9b52-5e7662b160c3\" (UID: \"088dcad0-68df-4eb4-9b52-5e7662b160c3\") " Nov 02 20:58:15.340827 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.340795 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088dcad0-68df-4eb4-9b52-5e7662b160c3-plugins-config-volume" pod "088dcad0-68df-4eb4-9b52-5e7662b160c3" (UID: "088dcad0-68df-4eb4-9b52-5e7662b160c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:58:15.340973 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.340959 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088dcad0-68df-4eb4-9b52-5e7662b160c3-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:58:15.341027 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.340977 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fjv9\" (UniqueName: \"kubernetes.io/projected/0e3e88e8-c62a-4e11-a732-793d093ef827-kube-api-access-2fjv9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:58:15.342727 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.342702 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088dcad0-68df-4eb4-9b52-5e7662b160c3-kube-api-access-frdps" pod "088dcad0-68df-4eb4-9b52-5e7662b160c3" (UID: "088dcad0-68df-4eb4-9b52-5e7662b160c3"). InnerVolumeSpecName "kube-api-access-frdps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:58:15.441971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.441887 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdps\" (UniqueName: \"kubernetes.io/projected/088dcad0-68df-4eb4-9b52-5e7662b160c3-kube-api-access-frdps\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:58:15.783355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.783316 5800 generic.go:356] "Generic (PLEG): container finished" podID="088dcad0-68df-4eb4-9b52-5e7662b160c3" containerID="5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe" exitCode=0 Nov 02 20:58:15.783355 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.783358 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" Nov 02 20:58:15.783655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.783402 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" event={"ID":"088dcad0-68df-4eb4-9b52-5e7662b160c3","Type":"ContainerDied","Data":"5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe"} Nov 02 20:58:15.783655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.783464 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6" event={"ID":"088dcad0-68df-4eb4-9b52-5e7662b160c3","Type":"ContainerDied","Data":"f6ac9a3450c1d9e1d2eb3a1f9dc577d64deb2d0f75be321355cea342ead0c059"} Nov 02 20:58:15.783655 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.783494 5800 scope.go:122] "RemoveContainer" containerID="5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe" Nov 02 20:58:15.785143 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.785113 5800 generic.go:356] "Generic (PLEG): container finished" podID="0e3e88e8-c62a-4e11-a732-793d093ef827" containerID="1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc" exitCode=0 Nov 02 20:58:15.785265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.785156 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" Nov 02 20:58:15.785265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.785207 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" event={"ID":"0e3e88e8-c62a-4e11-a732-793d093ef827","Type":"ContainerDied","Data":"1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc"} Nov 02 20:58:15.785265 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.785258 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv" event={"ID":"0e3e88e8-c62a-4e11-a732-793d093ef827","Type":"ContainerDied","Data":"af007d5e81adab6bb01cc8b1eb4272ccc660786fcef3f2d710597a76b2ea8fd2"} Nov 02 20:58:15.794709 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.794690 5800 scope.go:122] "RemoveContainer" containerID="5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe" Nov 02 20:58:15.795024 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:58:15.795004 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe\": container with ID starting with 5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe not found: ID does not exist" containerID="5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe" Nov 02 20:58:15.795076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.795031 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe"} err="failed to get container status \"5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe\": rpc error: code = NotFound desc = could not find container \"5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe\": container with ID starting with 5302f31060430855934978b6418f571d55600891e30e2a4ac6c9b295ff9f8bbe not found: ID does not exist" Nov 02 20:58:15.795076 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.795046 5800 scope.go:122] "RemoveContainer" containerID="1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc" Nov 02 20:58:15.804566 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.804548 5800 scope.go:122] "RemoveContainer" containerID="1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc" Nov 02 20:58:15.804815 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:58:15.804796 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc\": container with ID starting with 1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc not found: ID does not exist" containerID="1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc" Nov 02 20:58:15.804869 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.804821 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc"} err="failed to get container status \"1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc\": rpc error: code = NotFound desc = could not find container \"1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc\": container with ID starting with 1408d2d59017c580c37f73ba40a05c4370a8332f1f31d0c84c5f5b2fab1ee8fc not found: ID does not exist" Nov 02 20:58:15.937155 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.937126 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv"] Nov 02 20:58:15.985507 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:15.985473 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1dd5d5-585fc497cf-47fwv"] Nov 02 20:58:16.085576 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:16.085470 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6"] Nov 02 20:58:16.088879 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:16.088851 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1dd5d5-b88457d7-2dth6"] Nov 02 20:58:17.625368 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:17.625330 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088dcad0-68df-4eb4-9b52-5e7662b160c3" path="/var/lib/kubelet/pods/088dcad0-68df-4eb4-9b52-5e7662b160c3/volumes" Nov 02 20:58:17.625981 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:17.625962 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3e88e8-c62a-4e11-a732-793d093ef827" path="/var/lib/kubelet/pods/0e3e88e8-c62a-4e11-a732-793d093ef827/volumes" Nov 02 20:58:22.441043 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:22.441005 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:58:22.443034 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:22.443007 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 20:58:22.698522 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:58:22.698441 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:58:24.937412 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:24.937367 5800 scope.go:122] "RemoveContainer" containerID="eb61a26bdafdcb56f4bffbc21aa94bba520290482708893ab2328df6e622eff8" Nov 02 20:58:24.951184 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:58:24.951159 5800 scope.go:122] "RemoveContainer" containerID="51b87207758e2a3ab575a9674421c61b5bbbdd73dad88b7dbd409450b120abef" Nov 02 20:58:52.696872 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:58:52.696837 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:59:08.050708 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:08.050628 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc"] Nov 02 20:59:08.096643 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:08.096600 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae129e97-pqnfc"] Nov 02 20:59:09.635534 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:09.635495 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bcbe8fd-e744-4877-9c8c-7462dac7bd2f" path="/var/lib/kubelet/pods/5bcbe8fd-e744-4877-9c8c-7462dac7bd2f/volumes" Nov 02 20:59:11.043414 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:11.043379 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5"] Nov 02 20:59:11.089116 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:11.089077 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e8738b1-f7xr5"] Nov 02 20:59:11.624399 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:11.624366 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85020637-1049-484f-9c04-f402db4a8e58" path="/var/lib/kubelet/pods/85020637-1049-484f-9c04-f402db4a8e58/volumes" Nov 02 20:59:22.701194 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:59:22.701157 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 20:59:23.328430 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:23.328354 5800 generic.go:356] "Generic (PLEG): container finished" podID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerID="baa17535f2aa6e65bf7de744efbe3b8c77f876d0c4349087b50ab2eed5076dcd" exitCode=0 Nov 02 20:59:23.328621 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:23.328436 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" event={"ID":"e096d67b-03cb-45da-b019-d36fe8dc3f97","Type":"ContainerDied","Data":"baa17535f2aa6e65bf7de744efbe3b8c77f876d0c4349087b50ab2eed5076dcd"} Nov 02 20:59:23.328971 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:23.328954 5800 scope.go:122] "RemoveContainer" containerID="baa17535f2aa6e65bf7de744efbe3b8c77f876d0c4349087b50ab2eed5076dcd" Nov 02 20:59:23.330756 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:23.330731 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:59:23.330832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:23.330796 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:59:23.330832 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:23.330829 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:59:23.343123 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:23.343102 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:59:24.363381 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:24.363333 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" event={"ID":"e096d67b-03cb-45da-b019-d36fe8dc3f97","Type":"ContainerStarted","Data":"38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81"} Nov 02 20:59:25.001690 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:25.001647 5800 scope.go:122] "RemoveContainer" containerID="0ed3e7df7b6ee5092871ea02190eb6417d605f58ea686ca01c40bc66e2f9b043" Nov 02 20:59:25.221107 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:25.221060 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:59:25.222763 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:25.222731 5800 patch_prober.go:29] interesting pod/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 02 20:59:25.222893 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:25.222787 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 02 20:59:26.220891 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:26.220852 5800 patch_prober.go:29] interesting pod/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 02 20:59:26.221352 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:26.220910 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 02 20:59:27.220661 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:27.220624 5800 patch_prober.go:29] interesting pod/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 02 20:59:27.220855 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:27.220681 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 02 20:59:28.220593 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:28.220552 5800 patch_prober.go:29] interesting pod/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 02 20:59:28.220997 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:28.220608 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 02 20:59:29.220421 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:29.220388 5800 patch_prober.go:29] interesting pod/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" start-of-body= Nov 02 20:59:29.220620 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:29.220471 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.49:15021/healthz/ready\": dial tcp 10.132.1.49:15021: connect: connection refused" Nov 02 20:59:30.226494 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:30.226461 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:59:30.226938 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:30.226876 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:59:30.227656 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:30.227636 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 20:59:42.279324 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:42.279269 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6"] Nov 02 20:59:42.279937 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:42.279553 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" containerID="cri-o://e5ddd20311d649b456a492fa3f4ebf58901ec631dfca6da2c6c9ae17a9e5d680" gracePeriod=30 Nov 02 20:59:47.540124 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.540091 5800 generic.go:356] "Generic (PLEG): container finished" podID="ac776128-7547-4bde-875d-aa7abd33d23b" containerID="e5ddd20311d649b456a492fa3f4ebf58901ec631dfca6da2c6c9ae17a9e5d680" exitCode=0 Nov 02 20:59:47.540586 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.540138 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" event={"ID":"ac776128-7547-4bde-875d-aa7abd33d23b","Type":"ContainerDied","Data":"e5ddd20311d649b456a492fa3f4ebf58901ec631dfca6da2c6c9ae17a9e5d680"} Nov 02 20:59:47.585020 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.584990 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:59:47.705526 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705495 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-socket\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705547 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac776128-7547-4bde-875d-aa7abd33d23b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac776128-7547-4bde-875d-aa7abd33d23b-istiod-ca-cert\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705583 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-credential-socket\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705610 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-certs\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705731 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705670 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac776128-7547-4bde-875d-aa7abd33d23b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac776128-7547-4bde-875d-aa7abd33d23b-istio-podinfo\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705739 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-istio-token\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-istio-token\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705766 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-envoy\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705796 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-kube-api-access-v6gjv\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-kube-api-access-v6gjv\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.705994 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705846 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-data\") pod \"ac776128-7547-4bde-875d-aa7abd33d23b\" (UID: \"ac776128-7547-4bde-875d-aa7abd33d23b\") " Nov 02 20:59:47.707079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705792 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-socket" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:47.707079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.705909 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-credential-socket" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:47.707079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.706040 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac776128-7547-4bde-875d-aa7abd33d23b-istiod-ca-cert" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:59:47.707079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.706247 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-certs" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:47.707079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.706541 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-data" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:47.707079 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.707062 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.707578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.707087 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac776128-7547-4bde-875d-aa7abd33d23b-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.707578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.707106 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.707578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.707122 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.707578 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.707133 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.708296 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.708263 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac776128-7547-4bde-875d-aa7abd33d23b-istio-podinfo" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:59:47.708513 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.708494 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-envoy" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:59:47.708684 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.708661 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-istio-token" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:47.708935 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.708918 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-kube-api-access-v6gjv" pod "ac776128-7547-4bde-875d-aa7abd33d23b" (UID: "ac776128-7547-4bde-875d-aa7abd33d23b"). InnerVolumeSpecName "kube-api-access-v6gjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:59:47.808046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.807969 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac776128-7547-4bde-875d-aa7abd33d23b-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.808046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.807994 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.808046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.808008 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac776128-7547-4bde-875d-aa7abd33d23b-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:47.808046 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:47.808021 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gjv\" (UniqueName: \"kubernetes.io/projected/ac776128-7547-4bde-875d-aa7abd33d23b-kube-api-access-v6gjv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:59:48.552433 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:48.552402 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" Nov 02 20:59:48.552874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:48.552406 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6" event={"ID":"ac776128-7547-4bde-875d-aa7abd33d23b","Type":"ContainerDied","Data":"2d62e4db6c43a8aca3de7c9a6cc8d37cd711c65cd154599b699f74de2d8c6c84"} Nov 02 20:59:48.552874 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:48.552547 5800 scope.go:122] "RemoveContainer" containerID="e5ddd20311d649b456a492fa3f4ebf58901ec631dfca6da2c6c9ae17a9e5d680" Nov 02 20:59:48.757561 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:48.757529 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6"] Nov 02 20:59:48.803840 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:48.803761 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-09ecd0-openshift-default-964b9476f-f7sg6"] Nov 02 20:59:49.624229 ip-10-0-134-13 kubenswrapper[5800]: I1102 20:59:49.624192 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" path="/var/lib/kubelet/pods/ac776128-7547-4bde-875d-aa7abd33d23b/volumes" Nov 02 20:59:52.696384 ip-10-0-134-13 kubenswrapper[5800]: E1102 20:59:52.696344 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:00:00.160227 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.160187 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj"] Nov 02 21:00:00.161865 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.161834 5800 cpu_manager.go:414] "containerMap: removing container" podUID="088dcad0-68df-4eb4-9b52-5e7662b160c3" containerName="payload-pre-processing" Nov 02 21:00:00.161865 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.161863 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088dcad0-68df-4eb4-9b52-5e7662b160c3" containerName="payload-pre-processing" Nov 02 21:00:00.162119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.161920 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" Nov 02 21:00:00.162119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.161928 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" Nov 02 21:00:00.162119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.161950 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0e3e88e8-c62a-4e11-a732-793d093ef827" containerName="payload-processing" Nov 02 21:00:00.162119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.161958 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3e88e8-c62a-4e11-a732-793d093ef827" containerName="payload-processing" Nov 02 21:00:00.162119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.162112 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3e88e8-c62a-4e11-a732-793d093ef827" containerName="payload-processing" Nov 02 21:00:00.162418 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.162132 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="088dcad0-68df-4eb4-9b52-5e7662b160c3" containerName="payload-pre-processing" Nov 02 21:00:00.162418 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.162150 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac776128-7547-4bde-875d-aa7abd33d23b" containerName="istio-proxy" Nov 02 21:00:00.166670 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.166632 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk"] Nov 02 21:00:00.166968 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.166940 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.172034 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.171997 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k"] Nov 02 21:00:00.172187 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.172150 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.177772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.177749 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.206930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.206898 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x"] Nov 02 21:00:00.211838 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.211814 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj"] Nov 02 21:00:00.212004 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.211947 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.216601 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.216581 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf"] Nov 02 21:00:00.216737 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.216711 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.221671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.221647 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv"] Nov 02 21:00:00.221798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.221778 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.235316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.227893 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc"] Nov 02 21:00:00.235316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.228936 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.235316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.233615 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh"] Nov 02 21:00:00.235316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.233714 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.239938 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.239787 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc"] Nov 02 21:00:00.240068 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.239906 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.245889 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.245864 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.249427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.249400 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:00:00.251854 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.251701 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:00:00.251854 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.251780 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:00:00.254182 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.254137 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn"] Nov 02 21:00:00.272329 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.270171 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.281677 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.281489 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8"] Nov 02 21:00:00.288874 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.288747 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx"] Nov 02 21:00:00.289398 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.289365 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.294406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.293147 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ph6r\" (UniqueName: \"kubernetes.io/projected/529e13a2-3e07-4af5-aaea-c3942a800589-kube-api-access-4ph6r\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.294406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.293224 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529e13a2-3e07-4af5-aaea-c3942a800589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.294406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.293330 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fbf823-6929-493a-911c-3404641c8863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.294406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.293358 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snptt\" (UniqueName: \"kubernetes.io/projected/4baedbb3-e6fa-4a67-8e7f-a27005a97712-kube-api-access-snptt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.294406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.293390 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvc2v\" (UniqueName: \"kubernetes.io/projected/02fbf823-6929-493a-911c-3404641c8863-kube-api-access-jvc2v\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.294406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.293505 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baedbb3-e6fa-4a67-8e7f-a27005a97712-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.294850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.294716 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk"] Nov 02 21:00:00.294850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.294816 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.297162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.297137 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj"] Nov 02 21:00:00.303698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.303674 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x"] Nov 02 21:00:00.307047 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.307000 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k"] Nov 02 21:00:00.314105 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.311085 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj"] Nov 02 21:00:00.321464 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.316216 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc"] Nov 02 21:00:00.372983 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.372955 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx"] Nov 02 21:00:00.379952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.379870 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn"] Nov 02 21:00:00.384527 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.384498 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc"] Nov 02 21:00:00.388723 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.388669 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv"] Nov 02 21:00:00.392072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.392046 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh"] Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396180 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkqmh\" (UniqueName: \"kubernetes.io/projected/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-kube-api-access-fkqmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396231 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwr8\" (UniqueName: \"kubernetes.io/projected/33550673-a1d4-4cb5-9467-a20cc86f3823-kube-api-access-5bwr8\") pod \"maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396268 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33550673-a1d4-4cb5-9467-a20cc86f3823-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396350 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65wg\" (UniqueName: \"kubernetes.io/projected/de5226d7-33f1-41a5-a919-1c1bfc55962c-kube-api-access-f65wg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396405 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ph6r\" (UniqueName: \"kubernetes.io/projected/529e13a2-3e07-4af5-aaea-c3942a800589-kube-api-access-4ph6r\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396432 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scpf\" (UniqueName: \"kubernetes.io/projected/17a63e09-b535-4190-9a7f-f72258817ebc-kube-api-access-5scpf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396477 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5226d7-33f1-41a5-a919-1c1bfc55962c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396501 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvn5\" (UniqueName: \"kubernetes.io/projected/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-kube-api-access-9rvn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396528 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a63e09-b535-4190-9a7f-f72258817ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396576 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529e13a2-3e07-4af5-aaea-c3942a800589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396629 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3839a4-e546-4eed-b834-a62cd8b1f41c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396665 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fbf823-6929-493a-911c-3404641c8863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396690 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snptt\" (UniqueName: \"kubernetes.io/projected/4baedbb3-e6fa-4a67-8e7f-a27005a97712-kube-api-access-snptt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396716 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396756 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvc2v\" (UniqueName: \"kubernetes.io/projected/02fbf823-6929-493a-911c-3404641c8863-kube-api-access-jvc2v\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396816 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5gs\" (UniqueName: \"kubernetes.io/projected/2f4090b1-df6d-4014-8524-f39154343d5b-kube-api-access-ql5gs\") pod \"maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396856 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396892 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbdf\" (UniqueName: \"kubernetes.io/projected/ccbf204a-de19-4517-bc00-7bcc299496b3-kube-api-access-tsbdf\") pod \"maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396948 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf204a-de19-4517-bc00-7bcc299496b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.396979 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4090b1-df6d-4014-8524-f39154343d5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.397003 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccp2z\" (UniqueName: \"kubernetes.io/projected/8b3839a4-e546-4eed-b834-a62cd8b1f41c-kube-api-access-ccp2z\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.397051 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baedbb3-e6fa-4a67-8e7f-a27005a97712-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.397100 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnh2m\" (UniqueName: \"kubernetes.io/projected/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-kube-api-access-qnh2m\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.397142 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db514a1-d913-4b14-8e2c-0770313c6d9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.397191 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqfg\" (UniqueName: \"kubernetes.io/projected/2db514a1-d913-4b14-8e2c-0770313c6d9d-kube-api-access-6bqfg\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.397230 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.398521 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529e13a2-3e07-4af5-aaea-c3942a800589-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.400492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.399191 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fbf823-6929-493a-911c-3404641c8863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.402917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.402894 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baedbb3-e6fa-4a67-8e7f-a27005a97712-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.406603 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.406569 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8"] Nov 02 21:00:00.411950 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.411892 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf"] Nov 02 21:00:00.450470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.450428 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ph6r\" (UniqueName: \"kubernetes.io/projected/529e13a2-3e07-4af5-aaea-c3942a800589-kube-api-access-4ph6r\") pod \"maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.455768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.455733 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snptt\" (UniqueName: \"kubernetes.io/projected/4baedbb3-e6fa-4a67-8e7f-a27005a97712-kube-api-access-snptt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.456007 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.455989 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvc2v\" (UniqueName: \"kubernetes.io/projected/02fbf823-6929-493a-911c-3404641c8863-kube-api-access-jvc2v\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.484646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.484614 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:00.495002 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.494978 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:00.498651 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498625 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65wg\" (UniqueName: \"kubernetes.io/projected/de5226d7-33f1-41a5-a919-1c1bfc55962c-kube-api-access-f65wg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.498752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498674 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scpf\" (UniqueName: \"kubernetes.io/projected/17a63e09-b535-4190-9a7f-f72258817ebc-kube-api-access-5scpf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.498752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498709 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5226d7-33f1-41a5-a919-1c1bfc55962c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.498752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498725 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvn5\" (UniqueName: \"kubernetes.io/projected/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-kube-api-access-9rvn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.498936 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498753 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a63e09-b535-4190-9a7f-f72258817ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.498936 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498818 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3839a4-e546-4eed-b834-a62cd8b1f41c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.498936 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498868 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.499108 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498934 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5gs\" (UniqueName: \"kubernetes.io/projected/2f4090b1-df6d-4014-8524-f39154343d5b-kube-api-access-ql5gs\") pod \"maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.499108 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.498968 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.499108 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499001 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbdf\" (UniqueName: \"kubernetes.io/projected/ccbf204a-de19-4517-bc00-7bcc299496b3-kube-api-access-tsbdf\") pod \"maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.499303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499174 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf204a-de19-4517-bc00-7bcc299496b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.499303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499212 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4090b1-df6d-4014-8524-f39154343d5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.499303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499241 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccp2z\" (UniqueName: \"kubernetes.io/projected/8b3839a4-e546-4eed-b834-a62cd8b1f41c-kube-api-access-ccp2z\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.499606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499361 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnh2m\" (UniqueName: \"kubernetes.io/projected/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-kube-api-access-qnh2m\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.499606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499404 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db514a1-d913-4b14-8e2c-0770313c6d9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.499606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499454 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqfg\" (UniqueName: \"kubernetes.io/projected/2db514a1-d913-4b14-8e2c-0770313c6d9d-kube-api-access-6bqfg\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.499606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499489 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.499606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499570 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkqmh\" (UniqueName: \"kubernetes.io/projected/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-kube-api-access-fkqmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.499606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499575 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5226d7-33f1-41a5-a919-1c1bfc55962c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.499949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499815 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwr8\" (UniqueName: \"kubernetes.io/projected/33550673-a1d4-4cb5-9467-a20cc86f3823-kube-api-access-5bwr8\") pod \"maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.499949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499867 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33550673-a1d4-4cb5-9467-a20cc86f3823-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.499949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.499926 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.500140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.500053 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.500140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.500065 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3839a4-e546-4eed-b834-a62cd8b1f41c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.500443 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.500331 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf204a-de19-4517-bc00-7bcc299496b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.501004 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.500916 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33550673-a1d4-4cb5-9467-a20cc86f3823-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.501194 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.501166 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db514a1-d913-4b14-8e2c-0770313c6d9d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.501517 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.501352 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4090b1-df6d-4014-8524-f39154343d5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.501634 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.501613 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.501708 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.501615 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a63e09-b535-4190-9a7f-f72258817ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.504251 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.504226 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:00.556230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.556003 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbdf\" (UniqueName: \"kubernetes.io/projected/ccbf204a-de19-4517-bc00-7bcc299496b3-kube-api-access-tsbdf\") pod \"maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.556230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.556172 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scpf\" (UniqueName: \"kubernetes.io/projected/17a63e09-b535-4190-9a7f-f72258817ebc-kube-api-access-5scpf\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.557027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.556636 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvn5\" (UniqueName: \"kubernetes.io/projected/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-kube-api-access-9rvn5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.557027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.556866 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:00.562947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.558931 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65wg\" (UniqueName: \"kubernetes.io/projected/de5226d7-33f1-41a5-a919-1c1bfc55962c-kube-api-access-f65wg\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.562947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.560813 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkqmh\" (UniqueName: \"kubernetes.io/projected/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-kube-api-access-fkqmh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.562947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.560822 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqfg\" (UniqueName: \"kubernetes.io/projected/2db514a1-d913-4b14-8e2c-0770313c6d9d-kube-api-access-6bqfg\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.562947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.561266 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5gs\" (UniqueName: \"kubernetes.io/projected/2f4090b1-df6d-4014-8524-f39154343d5b-kube-api-access-ql5gs\") pod \"maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.562947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.561952 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwr8\" (UniqueName: \"kubernetes.io/projected/33550673-a1d4-4cb5-9467-a20cc86f3823-kube-api-access-5bwr8\") pod \"maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.562947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.562904 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccp2z\" (UniqueName: \"kubernetes.io/projected/8b3839a4-e546-4eed-b834-a62cd8b1f41c-kube-api-access-ccp2z\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.563854 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.563811 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnh2m\" (UniqueName: \"kubernetes.io/projected/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-kube-api-access-qnh2m\") pod \"maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.580100 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.579734 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:00.600622 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.600591 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:00.632811 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.632775 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:00.643261 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.643232 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:00.677664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.676556 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:00.680351 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.678964 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:00.691470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.691425 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:00.832075 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.831747 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:00.846594 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.840360 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:00.972089 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:00.972054 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k"] Nov 02 21:00:01.101253 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.101221 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk"] Nov 02 21:00:01.103563 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:00:01.103528 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4baedbb3_e6fa_4a67_8e7f_a27005a97712.slice/crio-42e20dbabe0303021b73d5ad27397e4179817939de2426b8ac37d48409fa36e6 WatchSource:0}: Error finding container 42e20dbabe0303021b73d5ad27397e4179817939de2426b8ac37d48409fa36e6: Status 404 returned error can't find the container with id 42e20dbabe0303021b73d5ad27397e4179817939de2426b8ac37d48409fa36e6 Nov 02 21:00:01.107070 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.107044 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj"] Nov 02 21:00:01.207155 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.207126 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf"] Nov 02 21:00:01.231771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.231739 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv"] Nov 02 21:00:01.243761 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.243731 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc"] Nov 02 21:00:01.369237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.369209 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn"] Nov 02 21:00:01.373027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.373000 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh"] Nov 02 21:00:01.382736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.382712 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx"] Nov 02 21:00:01.421882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.421855 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8"] Nov 02 21:00:01.480138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.479964 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x"] Nov 02 21:00:01.492429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.492378 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc"] Nov 02 21:00:01.501047 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.501018 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj"] Nov 02 21:00:01.691221 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.691183 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" event={"ID":"529e13a2-3e07-4af5-aaea-c3942a800589","Type":"ContainerStarted","Data":"48f56745cd38f6d17f3d05768cf3d8f4e0c064c4b13ddbe1dfcee35aaf24277b"} Nov 02 21:00:01.691436 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.691249 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" event={"ID":"529e13a2-3e07-4af5-aaea-c3942a800589","Type":"ContainerStarted","Data":"bf900659da238b4c8bb58b9837e05ba47208ce2a3a5815d529c35ab67c5313d6"} Nov 02 21:00:01.692947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.692918 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" event={"ID":"4baedbb3-e6fa-4a67-8e7f-a27005a97712","Type":"ContainerStarted","Data":"bed4002984647ee42f8f918cfb5eef938e9d6aea405030ce0c1fbc4843e5bf6a"} Nov 02 21:00:01.693102 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.692969 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" event={"ID":"4baedbb3-e6fa-4a67-8e7f-a27005a97712","Type":"ContainerStarted","Data":"42e20dbabe0303021b73d5ad27397e4179817939de2426b8ac37d48409fa36e6"} Nov 02 21:00:01.694312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.694259 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" event={"ID":"7fe0e8ef-ce92-40c0-89d4-7c1d91018892","Type":"ContainerStarted","Data":"9ee391b7b275b223e6e31ca3bb1941825bd83053eafe0ca29b652be90d81e4a8"} Nov 02 21:00:01.695558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.695532 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerStarted","Data":"56d052deb24295696c6761a3748409223ebd5a32557cda7f0b265da37c232179"} Nov 02 21:00:01.696812 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.696791 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" event={"ID":"b37f3722-ebfe-4975-8b8f-6cd72a3c473e","Type":"ContainerStarted","Data":"dc366ceb3b789b14cd13204c4674f1ce0fa4b8428523dc343c220ed113827776"} Nov 02 21:00:01.698074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.698052 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" event={"ID":"de5226d7-33f1-41a5-a919-1c1bfc55962c","Type":"ContainerStarted","Data":"cc4d8588f5465e9cda31f80d9b43eaa5697ccc1e1a2d0eeec6fd0053f999ef28"} Nov 02 21:00:01.699302 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.699268 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" event={"ID":"8b3839a4-e546-4eed-b834-a62cd8b1f41c","Type":"ContainerStarted","Data":"229c3d4118e855f54a1eb581b9cab51567cb4b7e363a7baf4ec2830fd97c12e9"} Nov 02 21:00:01.700558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.700537 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerStarted","Data":"8b22ba83bd1dead8bad328c8b8d01d28e393307c148f3578e92c7220715ea246"} Nov 02 21:00:01.702117 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.702060 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" event={"ID":"02fbf823-6929-493a-911c-3404641c8863","Type":"ContainerStarted","Data":"5ed788eb3aef952abbab9849b81b25afe40b0e3190d8a6f4d72ac4685e4b0d85"} Nov 02 21:00:01.703681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.703657 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerStarted","Data":"184082b18dab245c4256456246dd7a2915e33eea91c882a73892faccf82515eb"} Nov 02 21:00:01.704870 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.704848 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerStarted","Data":"b508aba616e5e1e0dd9601f923ca79744dc749a4924636194bbff7a4a7ecdbc3"} Nov 02 21:00:01.706101 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.706070 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" event={"ID":"2f4090b1-df6d-4014-8524-f39154343d5b","Type":"ContainerStarted","Data":"df3650456e538197173e489d4f2a440f839ca2fd2c1697b63ec02eda6a26267a"} Nov 02 21:00:01.707347 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.707328 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" event={"ID":"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939","Type":"ContainerStarted","Data":"b485c7d4205309e7735051449fcba3a7ef1dc982c660fc0a48944d9e2fd95a9a"} Nov 02 21:00:01.804166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.804103 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" podStartSLOduration=1.804081233 podStartE2EDuration="1.804081233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:01.800725088 +0000 UTC m=+4300.317497674" watchObservedRunningTime="2026-11-02 21:00:01.804081233 +0000 UTC m=+4300.320853819" Nov 02 21:00:01.906729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:01.906671 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" podStartSLOduration=1.906654759 podStartE2EDuration="1.906654759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:01.905569475 +0000 UTC m=+4300.422342062" watchObservedRunningTime="2026-11-02 21:00:01.906654759 +0000 UTC m=+4300.423427344" Nov 02 21:00:02.717890 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.717854 5800 generic.go:356] "Generic (PLEG): container finished" podID="2f4090b1-df6d-4014-8524-f39154343d5b" containerID="4bef9227a96e91d2df4c64fbc2e0849e59578fde5866e62a0dccba9f99134d53" exitCode=60 Nov 02 21:00:02.718412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.717934 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" event={"ID":"2f4090b1-df6d-4014-8524-f39154343d5b","Type":"ContainerDied","Data":"4bef9227a96e91d2df4c64fbc2e0849e59578fde5866e62a0dccba9f99134d53"} Nov 02 21:00:02.718412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.718320 5800 scope.go:122] "RemoveContainer" containerID="4bef9227a96e91d2df4c64fbc2e0849e59578fde5866e62a0dccba9f99134d53" Nov 02 21:00:02.719777 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.719750 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" event={"ID":"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939","Type":"ContainerStarted","Data":"27a8115c5955eb28d50ce0c7263bff8b66fa06eabab101fba5828be4764c6dff"} Nov 02 21:00:02.721500 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.721474 5800 generic.go:356] "Generic (PLEG): container finished" podID="529e13a2-3e07-4af5-aaea-c3942a800589" containerID="48f56745cd38f6d17f3d05768cf3d8f4e0c064c4b13ddbe1dfcee35aaf24277b" exitCode=60 Nov 02 21:00:02.721614 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.721594 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" event={"ID":"529e13a2-3e07-4af5-aaea-c3942a800589","Type":"ContainerDied","Data":"48f56745cd38f6d17f3d05768cf3d8f4e0c064c4b13ddbe1dfcee35aaf24277b"} Nov 02 21:00:02.721848 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.721817 5800 scope.go:122] "RemoveContainer" containerID="48f56745cd38f6d17f3d05768cf3d8f4e0c064c4b13ddbe1dfcee35aaf24277b" Nov 02 21:00:02.723822 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.723798 5800 generic.go:356] "Generic (PLEG): container finished" podID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerID="bed4002984647ee42f8f918cfb5eef938e9d6aea405030ce0c1fbc4843e5bf6a" exitCode=60 Nov 02 21:00:02.723938 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.723882 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" event={"ID":"4baedbb3-e6fa-4a67-8e7f-a27005a97712","Type":"ContainerDied","Data":"bed4002984647ee42f8f918cfb5eef938e9d6aea405030ce0c1fbc4843e5bf6a"} Nov 02 21:00:02.724104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.724087 5800 scope.go:122] "RemoveContainer" containerID="bed4002984647ee42f8f918cfb5eef938e9d6aea405030ce0c1fbc4843e5bf6a" Nov 02 21:00:02.732230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.732116 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" event={"ID":"7fe0e8ef-ce92-40c0-89d4-7c1d91018892","Type":"ContainerStarted","Data":"db8e5603f91a412d3a9f7c55713c401de06833c6c947b1d4e0a9e6616cd22810"} Nov 02 21:00:02.736555 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.736505 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerStarted","Data":"a0b90b4a9b83a4fa43ba2029437b1cb68bd745b35fd10907f68fe6616b69454b"} Nov 02 21:00:02.750537 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.750507 5800 generic.go:356] "Generic (PLEG): container finished" podID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerID="2d85416e1397f5a5466db0fa6a6c1ed28b337e7b0bce7a4c2e4049b7ef2aa322" exitCode=60 Nov 02 21:00:02.750815 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.750587 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" event={"ID":"b37f3722-ebfe-4975-8b8f-6cd72a3c473e","Type":"ContainerDied","Data":"2d85416e1397f5a5466db0fa6a6c1ed28b337e7b0bce7a4c2e4049b7ef2aa322"} Nov 02 21:00:02.750908 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.750834 5800 scope.go:122] "RemoveContainer" containerID="2d85416e1397f5a5466db0fa6a6c1ed28b337e7b0bce7a4c2e4049b7ef2aa322" Nov 02 21:00:02.758460 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.758170 5800 generic.go:356] "Generic (PLEG): container finished" podID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerID="1016b0889338d6d002603e2b10e9b74b8c9afeed2ffb98c0c6e1bb735a51712e" exitCode=60 Nov 02 21:00:02.758718 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.758615 5800 scope.go:122] "RemoveContainer" containerID="1016b0889338d6d002603e2b10e9b74b8c9afeed2ffb98c0c6e1bb735a51712e" Nov 02 21:00:02.758944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.758921 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" event={"ID":"de5226d7-33f1-41a5-a919-1c1bfc55962c","Type":"ContainerDied","Data":"1016b0889338d6d002603e2b10e9b74b8c9afeed2ffb98c0c6e1bb735a51712e"} Nov 02 21:00:02.761040 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.761016 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" event={"ID":"8b3839a4-e546-4eed-b834-a62cd8b1f41c","Type":"ContainerStarted","Data":"6cabaf6d81b0f9482e33fe65572f06876c44199291b4a6b3f4d7a7d7d9a1c169"} Nov 02 21:00:02.763412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.763006 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerStarted","Data":"26144ebc75fb3204681fcdf7df030025e220b1702d9ce166f8cab65f328cb9a5"} Nov 02 21:00:02.765800 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.765751 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" event={"ID":"02fbf823-6929-493a-911c-3404641c8863","Type":"ContainerDied","Data":"8e66c388a1ff8b0112a2ff17d54f9654dec36c7175ae6652936d96584fbe6b43"} Nov 02 21:00:02.767385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.767244 5800 generic.go:356] "Generic (PLEG): container finished" podID="02fbf823-6929-493a-911c-3404641c8863" containerID="8e66c388a1ff8b0112a2ff17d54f9654dec36c7175ae6652936d96584fbe6b43" exitCode=60 Nov 02 21:00:02.769670 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.768776 5800 scope.go:122] "RemoveContainer" containerID="8e66c388a1ff8b0112a2ff17d54f9654dec36c7175ae6652936d96584fbe6b43" Nov 02 21:00:02.777585 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.777555 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerStarted","Data":"08627a3e309c26a7c5bdd9868264c06fdbe4292aa35e4b170aab50bf39453fed"} Nov 02 21:00:02.782610 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:02.782573 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerStarted","Data":"09224d2cfe4ce546947eaeec37e573d0e8941eee83ae834600f10328c48a9652"} Nov 02 21:00:03.147295 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.147229 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" podStartSLOduration=3.147216271 podStartE2EDuration="3.147216271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.146867891 +0000 UTC m=+4301.663640475" watchObservedRunningTime="2026-11-02 21:00:03.147216271 +0000 UTC m=+4301.663988849" Nov 02 21:00:03.224988 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:03.224953 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0d3f5d_938b_4bc1_b4fa_6c8d168fe939.slice/crio-27a8115c5955eb28d50ce0c7263bff8b66fa06eabab101fba5828be4764c6dff.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:00:03.359561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.359493 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" podStartSLOduration=3.359475684 podStartE2EDuration="3.359475684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.356307559 +0000 UTC m=+4301.873080136" watchObservedRunningTime="2026-11-02 21:00:03.359475684 +0000 UTC m=+4301.876248270" Nov 02 21:00:03.465074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.464974 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" podStartSLOduration=3.464959538 podStartE2EDuration="3.464959538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.461869995 +0000 UTC m=+4301.978642585" watchObservedRunningTime="2026-11-02 21:00:03.464959538 +0000 UTC m=+4301.981732123" Nov 02 21:00:03.572493 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.572439 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" podStartSLOduration=3.572423017 podStartE2EDuration="3.572423017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.569988396 +0000 UTC m=+4302.086760974" watchObservedRunningTime="2026-11-02 21:00:03.572423017 +0000 UTC m=+4302.089195603" Nov 02 21:00:03.675428 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.675378 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" podStartSLOduration=3.675364332 podStartE2EDuration="3.675364332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.674559841 +0000 UTC m=+4302.191332427" watchObservedRunningTime="2026-11-02 21:00:03.675364332 +0000 UTC m=+4302.192136916" Nov 02 21:00:03.793453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.793420 5800 generic.go:356] "Generic (PLEG): container finished" podID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerID="09224d2cfe4ce546947eaeec37e573d0e8941eee83ae834600f10328c48a9652" exitCode=60 Nov 02 21:00:03.793842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.793505 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerDied","Data":"09224d2cfe4ce546947eaeec37e573d0e8941eee83ae834600f10328c48a9652"} Nov 02 21:00:03.793842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.793826 5800 scope.go:122] "RemoveContainer" containerID="09224d2cfe4ce546947eaeec37e573d0e8941eee83ae834600f10328c48a9652" Nov 02 21:00:03.795710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.795691 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" event={"ID":"2f4090b1-df6d-4014-8524-f39154343d5b","Type":"ContainerStarted","Data":"1dc2569fdb9313eb50e30ec685de41f6f3e03e15f04df18da1695c742e1d1cb8"} Nov 02 21:00:03.797444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.797415 5800 generic.go:356] "Generic (PLEG): container finished" podID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerID="27a8115c5955eb28d50ce0c7263bff8b66fa06eabab101fba5828be4764c6dff" exitCode=60 Nov 02 21:00:03.797522 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.797494 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" event={"ID":"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939","Type":"ContainerDied","Data":"27a8115c5955eb28d50ce0c7263bff8b66fa06eabab101fba5828be4764c6dff"} Nov 02 21:00:03.797682 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.797666 5800 scope.go:122] "RemoveContainer" containerID="27a8115c5955eb28d50ce0c7263bff8b66fa06eabab101fba5828be4764c6dff" Nov 02 21:00:03.799575 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.799550 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" event={"ID":"529e13a2-3e07-4af5-aaea-c3942a800589","Type":"ContainerStarted","Data":"dff908de3796e2d92b32113f93d86d94c2b4911aeab9e5d9330e6574d2f5091d"} Nov 02 21:00:03.801560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.801537 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" event={"ID":"4baedbb3-e6fa-4a67-8e7f-a27005a97712","Type":"ContainerStarted","Data":"9125f61cedc93f4162d040ee04fef1189fb39b1d4e95a67659fa95945f5a34ec"} Nov 02 21:00:03.803098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.803073 5800 generic.go:356] "Generic (PLEG): container finished" podID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerID="db8e5603f91a412d3a9f7c55713c401de06833c6c947b1d4e0a9e6616cd22810" exitCode=60 Nov 02 21:00:03.803212 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.803146 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" event={"ID":"7fe0e8ef-ce92-40c0-89d4-7c1d91018892","Type":"ContainerDied","Data":"db8e5603f91a412d3a9f7c55713c401de06833c6c947b1d4e0a9e6616cd22810"} Nov 02 21:00:03.803348 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.803332 5800 scope.go:122] "RemoveContainer" containerID="db8e5603f91a412d3a9f7c55713c401de06833c6c947b1d4e0a9e6616cd22810" Nov 02 21:00:03.804807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.804787 5800 generic.go:356] "Generic (PLEG): container finished" podID="17a63e09-b535-4190-9a7f-f72258817ebc" containerID="a0b90b4a9b83a4fa43ba2029437b1cb68bd745b35fd10907f68fe6616b69454b" exitCode=60 Nov 02 21:00:03.804919 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.804864 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerDied","Data":"a0b90b4a9b83a4fa43ba2029437b1cb68bd745b35fd10907f68fe6616b69454b"} Nov 02 21:00:03.805037 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.805019 5800 scope.go:122] "RemoveContainer" containerID="a0b90b4a9b83a4fa43ba2029437b1cb68bd745b35fd10907f68fe6616b69454b" Nov 02 21:00:03.807493 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.807468 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" event={"ID":"b37f3722-ebfe-4975-8b8f-6cd72a3c473e","Type":"ContainerStarted","Data":"a35cf1d346831c9a793a408c3406b33ab8adf9a4b9b283c62843c1c28a67c801"} Nov 02 21:00:03.813496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.813471 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" event={"ID":"de5226d7-33f1-41a5-a919-1c1bfc55962c","Type":"ContainerStarted","Data":"1125ae4b6ceb468f5fcea077d89907c89ed112d9795ca2961a086b013485cb4a"} Nov 02 21:00:03.816383 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.816362 5800 generic.go:356] "Generic (PLEG): container finished" podID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerID="6cabaf6d81b0f9482e33fe65572f06876c44199291b4a6b3f4d7a7d7d9a1c169" exitCode=60 Nov 02 21:00:03.816489 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.816440 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" event={"ID":"8b3839a4-e546-4eed-b834-a62cd8b1f41c","Type":"ContainerDied","Data":"6cabaf6d81b0f9482e33fe65572f06876c44199291b4a6b3f4d7a7d7d9a1c169"} Nov 02 21:00:03.816669 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.816649 5800 scope.go:122] "RemoveContainer" containerID="6cabaf6d81b0f9482e33fe65572f06876c44199291b4a6b3f4d7a7d7d9a1c169" Nov 02 21:00:03.818161 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.818138 5800 generic.go:356] "Generic (PLEG): container finished" podID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerID="26144ebc75fb3204681fcdf7df030025e220b1702d9ce166f8cab65f328cb9a5" exitCode=60 Nov 02 21:00:03.818245 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.818225 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerDied","Data":"26144ebc75fb3204681fcdf7df030025e220b1702d9ce166f8cab65f328cb9a5"} Nov 02 21:00:03.818450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.818433 5800 scope.go:122] "RemoveContainer" containerID="26144ebc75fb3204681fcdf7df030025e220b1702d9ce166f8cab65f328cb9a5" Nov 02 21:00:03.820326 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.820298 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" event={"ID":"02fbf823-6929-493a-911c-3404641c8863","Type":"ContainerStarted","Data":"4d8c69f0f0837324b6e41927f7203057dddb33b1a2fc25549696934f149d0012"} Nov 02 21:00:03.822647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.822624 5800 generic.go:356] "Generic (PLEG): container finished" podID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerID="08627a3e309c26a7c5bdd9868264c06fdbe4292aa35e4b170aab50bf39453fed" exitCode=60 Nov 02 21:00:03.822759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.822667 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerDied","Data":"08627a3e309c26a7c5bdd9868264c06fdbe4292aa35e4b170aab50bf39453fed"} Nov 02 21:00:03.822898 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.822874 5800 scope.go:122] "RemoveContainer" containerID="08627a3e309c26a7c5bdd9868264c06fdbe4292aa35e4b170aab50bf39453fed" Nov 02 21:00:03.888754 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.888693 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" podStartSLOduration=3.8886754850000003 podStartE2EDuration="3.888675485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.886615527 +0000 UTC m=+4302.403388112" watchObservedRunningTime="2026-11-02 21:00:03.888675485 +0000 UTC m=+4302.405448071" Nov 02 21:00:03.997124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:03.997015 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" podStartSLOduration=3.997001457 podStartE2EDuration="3.997001457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:03.993912324 +0000 UTC m=+4302.510684911" watchObservedRunningTime="2026-11-02 21:00:03.997001457 +0000 UTC m=+4302.513774094" Nov 02 21:00:04.843623 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.843528 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" event={"ID":"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939","Type":"ContainerStarted","Data":"05e88ab3a6120c1ccb0a47e85b354bf5908141b76d7aff06d7f3597f4ce5e41d"} Nov 02 21:00:04.847453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.846644 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" podStartSLOduration=4.8466276839999995 podStartE2EDuration="4.846627684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:04.840115232 +0000 UTC m=+4303.356887814" watchObservedRunningTime="2026-11-02 21:00:04.846627684 +0000 UTC m=+4303.363400272" Nov 02 21:00:04.847453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.846699 5800 generic.go:356] "Generic (PLEG): container finished" podID="529e13a2-3e07-4af5-aaea-c3942a800589" containerID="dff908de3796e2d92b32113f93d86d94c2b4911aeab9e5d9330e6574d2f5091d" exitCode=60 Nov 02 21:00:04.847453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.846865 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" event={"ID":"529e13a2-3e07-4af5-aaea-c3942a800589","Type":"ContainerDied","Data":"dff908de3796e2d92b32113f93d86d94c2b4911aeab9e5d9330e6574d2f5091d"} Nov 02 21:00:04.847453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.846935 5800 scope.go:122] "RemoveContainer" containerID="48f56745cd38f6d17f3d05768cf3d8f4e0c064c4b13ddbe1dfcee35aaf24277b" Nov 02 21:00:04.847453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.847299 5800 scope.go:122] "RemoveContainer" containerID="dff908de3796e2d92b32113f93d86d94c2b4911aeab9e5d9330e6574d2f5091d" Nov 02 21:00:04.848073 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:04.847979 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k_odh-ai-gateway-infra(529e13a2-3e07-4af5-aaea-c3942a800589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" Nov 02 21:00:04.850611 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.850586 5800 generic.go:356] "Generic (PLEG): container finished" podID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerID="9125f61cedc93f4162d040ee04fef1189fb39b1d4e95a67659fa95945f5a34ec" exitCode=60 Nov 02 21:00:04.850729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.850611 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" event={"ID":"4baedbb3-e6fa-4a67-8e7f-a27005a97712","Type":"ContainerDied","Data":"9125f61cedc93f4162d040ee04fef1189fb39b1d4e95a67659fa95945f5a34ec"} Nov 02 21:00:04.850962 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.850944 5800 scope.go:122] "RemoveContainer" containerID="9125f61cedc93f4162d040ee04fef1189fb39b1d4e95a67659fa95945f5a34ec" Nov 02 21:00:04.851340 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:04.851313 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk_odh-ai-gateway-infra(4baedbb3-e6fa-4a67-8e7f-a27005a97712)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" Nov 02 21:00:04.852812 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.852789 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" event={"ID":"7fe0e8ef-ce92-40c0-89d4-7c1d91018892","Type":"ContainerStarted","Data":"511b1f0d4861dd4ec0dc7fcc2d82c79d6f7ec18ea7f53d2299829507677e0233"} Nov 02 21:00:04.854920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.854899 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerStarted","Data":"074f5588b0ea423309b1820cc799bd32e84d8b0df1356beb40f430a9d306b03a"} Nov 02 21:00:04.856769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.856749 5800 generic.go:356] "Generic (PLEG): container finished" podID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerID="a35cf1d346831c9a793a408c3406b33ab8adf9a4b9b283c62843c1c28a67c801" exitCode=60 Nov 02 21:00:04.856873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.856862 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" event={"ID":"b37f3722-ebfe-4975-8b8f-6cd72a3c473e","Type":"ContainerDied","Data":"a35cf1d346831c9a793a408c3406b33ab8adf9a4b9b283c62843c1c28a67c801"} Nov 02 21:00:04.857115 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.857102 5800 scope.go:122] "RemoveContainer" containerID="a35cf1d346831c9a793a408c3406b33ab8adf9a4b9b283c62843c1c28a67c801" Nov 02 21:00:04.857357 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:04.857340 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf_odh-ai-gateway-infra(b37f3722-ebfe-4975-8b8f-6cd72a3c473e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" Nov 02 21:00:04.859330 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.859265 5800 generic.go:356] "Generic (PLEG): container finished" podID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerID="1125ae4b6ceb468f5fcea077d89907c89ed112d9795ca2961a086b013485cb4a" exitCode=60 Nov 02 21:00:04.859412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.859307 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" event={"ID":"de5226d7-33f1-41a5-a919-1c1bfc55962c","Type":"ContainerDied","Data":"1125ae4b6ceb468f5fcea077d89907c89ed112d9795ca2961a086b013485cb4a"} Nov 02 21:00:04.859520 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.859506 5800 scope.go:122] "RemoveContainer" containerID="1125ae4b6ceb468f5fcea077d89907c89ed112d9795ca2961a086b013485cb4a" Nov 02 21:00:04.859733 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:04.859707 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc_odh-ai-gateway-infra(de5226d7-33f1-41a5-a919-1c1bfc55962c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" Nov 02 21:00:04.861445 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.861425 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" event={"ID":"8b3839a4-e546-4eed-b834-a62cd8b1f41c","Type":"ContainerStarted","Data":"32a6a571370d331fc27598b5f555efa82695c9a800cec2a95fca2c42c2ca7b80"} Nov 02 21:00:04.863407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.863388 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerStarted","Data":"df61a025fa78b1dc6cf41199bbb0e9e6ad56f2abdf74701d2a553c536ffad6f4"} Nov 02 21:00:04.865324 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.865272 5800 scope.go:122] "RemoveContainer" containerID="bed4002984647ee42f8f918cfb5eef938e9d6aea405030ce0c1fbc4843e5bf6a" Nov 02 21:00:04.865396 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.865342 5800 generic.go:356] "Generic (PLEG): container finished" podID="02fbf823-6929-493a-911c-3404641c8863" containerID="4d8c69f0f0837324b6e41927f7203057dddb33b1a2fc25549696934f149d0012" exitCode=60 Nov 02 21:00:04.865470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.865404 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" event={"ID":"02fbf823-6929-493a-911c-3404641c8863","Type":"ContainerDied","Data":"4d8c69f0f0837324b6e41927f7203057dddb33b1a2fc25549696934f149d0012"} Nov 02 21:00:04.865615 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.865595 5800 scope.go:122] "RemoveContainer" containerID="4d8c69f0f0837324b6e41927f7203057dddb33b1a2fc25549696934f149d0012" Nov 02 21:00:04.865843 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:04.865813 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj_odh-ai-gateway-infra(02fbf823-6929-493a-911c-3404641c8863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" podUID="02fbf823-6929-493a-911c-3404641c8863" Nov 02 21:00:04.867482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.867463 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerStarted","Data":"efbf1b476b5f39af801342891e24b40aca540b92a12097fb64cff41b7b119cba"} Nov 02 21:00:04.869384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.869363 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerStarted","Data":"8f2b9e97acb1c0b41e6d50c73903bc3e842dee69df3c6aec6d3e6eb589d75202"} Nov 02 21:00:04.871161 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.871142 5800 generic.go:356] "Generic (PLEG): container finished" podID="2f4090b1-df6d-4014-8524-f39154343d5b" containerID="1dc2569fdb9313eb50e30ec685de41f6f3e03e15f04df18da1695c742e1d1cb8" exitCode=60 Nov 02 21:00:04.871242 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.871195 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" event={"ID":"2f4090b1-df6d-4014-8524-f39154343d5b","Type":"ContainerDied","Data":"1dc2569fdb9313eb50e30ec685de41f6f3e03e15f04df18da1695c742e1d1cb8"} Nov 02 21:00:04.871466 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.871450 5800 scope.go:122] "RemoveContainer" containerID="1dc2569fdb9313eb50e30ec685de41f6f3e03e15f04df18da1695c742e1d1cb8" Nov 02 21:00:04.871754 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:04.871733 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv_odh-ai-gateway-infra(2f4090b1-df6d-4014-8524-f39154343d5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" Nov 02 21:00:04.880729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.880709 5800 scope.go:122] "RemoveContainer" containerID="2d85416e1397f5a5466db0fa6a6c1ed28b337e7b0bce7a4c2e4049b7ef2aa322" Nov 02 21:00:04.895411 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.895387 5800 scope.go:122] "RemoveContainer" containerID="1016b0889338d6d002603e2b10e9b74b8c9afeed2ffb98c0c6e1bb735a51712e" Nov 02 21:00:04.910375 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.910357 5800 scope.go:122] "RemoveContainer" containerID="8e66c388a1ff8b0112a2ff17d54f9654dec36c7175ae6652936d96584fbe6b43" Nov 02 21:00:04.925668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:04.925642 5800 scope.go:122] "RemoveContainer" containerID="4bef9227a96e91d2df4c64fbc2e0849e59578fde5866e62a0dccba9f99134d53" Nov 02 21:00:05.160598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.160539 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" podStartSLOduration=5.160522211 podStartE2EDuration="5.160522211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:05.155458715 +0000 UTC m=+4303.672231313" watchObservedRunningTime="2026-11-02 21:00:05.160522211 +0000 UTC m=+4303.677294797" Nov 02 21:00:05.262715 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.262653 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" podStartSLOduration=5.262639961 podStartE2EDuration="5.262639961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:05.25968706 +0000 UTC m=+4303.776459646" watchObservedRunningTime="2026-11-02 21:00:05.262639961 +0000 UTC m=+4303.779412606" Nov 02 21:00:05.367801 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.367744 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" podStartSLOduration=5.367729721 podStartE2EDuration="5.367729721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:05.363925145 +0000 UTC m=+4303.880697728" watchObservedRunningTime="2026-11-02 21:00:05.367729721 +0000 UTC m=+4303.884502324" Nov 02 21:00:05.882871 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.882836 5800 generic.go:356] "Generic (PLEG): container finished" podID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerID="8f2b9e97acb1c0b41e6d50c73903bc3e842dee69df3c6aec6d3e6eb589d75202" exitCode=60 Nov 02 21:00:05.883429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.882912 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerDied","Data":"8f2b9e97acb1c0b41e6d50c73903bc3e842dee69df3c6aec6d3e6eb589d75202"} Nov 02 21:00:05.883429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.882963 5800 scope.go:122] "RemoveContainer" containerID="09224d2cfe4ce546947eaeec37e573d0e8941eee83ae834600f10328c48a9652" Nov 02 21:00:05.883429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.883264 5800 scope.go:122] "RemoveContainer" containerID="8f2b9e97acb1c0b41e6d50c73903bc3e842dee69df3c6aec6d3e6eb589d75202" Nov 02 21:00:05.883700 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.883673 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj_odh-ai-gateway-infra(33550673-a1d4-4cb5-9467-a20cc86f3823)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" Nov 02 21:00:05.885668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.885646 5800 scope.go:122] "RemoveContainer" containerID="1dc2569fdb9313eb50e30ec685de41f6f3e03e15f04df18da1695c742e1d1cb8" Nov 02 21:00:05.885982 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.885959 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv_odh-ai-gateway-infra(2f4090b1-df6d-4014-8524-f39154343d5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" Nov 02 21:00:05.887691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.887668 5800 generic.go:356] "Generic (PLEG): container finished" podID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerID="05e88ab3a6120c1ccb0a47e85b354bf5908141b76d7aff06d7f3597f4ce5e41d" exitCode=60 Nov 02 21:00:05.887814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.887785 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" event={"ID":"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939","Type":"ContainerDied","Data":"05e88ab3a6120c1ccb0a47e85b354bf5908141b76d7aff06d7f3597f4ce5e41d"} Nov 02 21:00:05.888189 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.888167 5800 scope.go:122] "RemoveContainer" containerID="05e88ab3a6120c1ccb0a47e85b354bf5908141b76d7aff06d7f3597f4ce5e41d" Nov 02 21:00:05.888492 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.888467 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8_odh-ai-gateway-infra(5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" Nov 02 21:00:05.890011 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.889991 5800 scope.go:122] "RemoveContainer" containerID="dff908de3796e2d92b32113f93d86d94c2b4911aeab9e5d9330e6574d2f5091d" Nov 02 21:00:05.890275 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.890254 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k_odh-ai-gateway-infra(529e13a2-3e07-4af5-aaea-c3942a800589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" Nov 02 21:00:05.892265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.892241 5800 scope.go:122] "RemoveContainer" containerID="9125f61cedc93f4162d040ee04fef1189fb39b1d4e95a67659fa95945f5a34ec" Nov 02 21:00:05.892555 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.892519 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk_odh-ai-gateway-infra(4baedbb3-e6fa-4a67-8e7f-a27005a97712)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" Nov 02 21:00:05.900009 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.899992 5800 scope.go:122] "RemoveContainer" containerID="27a8115c5955eb28d50ce0c7263bff8b66fa06eabab101fba5828be4764c6dff" Nov 02 21:00:05.900733 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.900710 5800 generic.go:356] "Generic (PLEG): container finished" podID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerID="511b1f0d4861dd4ec0dc7fcc2d82c79d6f7ec18ea7f53d2299829507677e0233" exitCode=60 Nov 02 21:00:05.900825 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.900774 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" event={"ID":"7fe0e8ef-ce92-40c0-89d4-7c1d91018892","Type":"ContainerDied","Data":"511b1f0d4861dd4ec0dc7fcc2d82c79d6f7ec18ea7f53d2299829507677e0233"} Nov 02 21:00:05.901165 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.901145 5800 scope.go:122] "RemoveContainer" containerID="511b1f0d4861dd4ec0dc7fcc2d82c79d6f7ec18ea7f53d2299829507677e0233" Nov 02 21:00:05.901479 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.901456 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx_odh-ai-gateway-infra(7fe0e8ef-ce92-40c0-89d4-7c1d91018892)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" Nov 02 21:00:05.902732 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.902709 5800 generic.go:356] "Generic (PLEG): container finished" podID="17a63e09-b535-4190-9a7f-f72258817ebc" containerID="074f5588b0ea423309b1820cc799bd32e84d8b0df1356beb40f430a9d306b03a" exitCode=60 Nov 02 21:00:05.902815 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.902792 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerDied","Data":"074f5588b0ea423309b1820cc799bd32e84d8b0df1356beb40f430a9d306b03a"} Nov 02 21:00:05.902985 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.902971 5800 scope.go:122] "RemoveContainer" containerID="074f5588b0ea423309b1820cc799bd32e84d8b0df1356beb40f430a9d306b03a" Nov 02 21:00:05.903192 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.903171 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x_odh-ai-gateway-infra(17a63e09-b535-4190-9a7f-f72258817ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" Nov 02 21:00:05.904649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.904630 5800 scope.go:122] "RemoveContainer" containerID="a35cf1d346831c9a793a408c3406b33ab8adf9a4b9b283c62843c1c28a67c801" Nov 02 21:00:05.904894 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.904872 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf_odh-ai-gateway-infra(b37f3722-ebfe-4975-8b8f-6cd72a3c473e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" Nov 02 21:00:05.906317 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.906296 5800 scope.go:122] "RemoveContainer" containerID="1125ae4b6ceb468f5fcea077d89907c89ed112d9795ca2961a086b013485cb4a" Nov 02 21:00:05.906542 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.906519 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc_odh-ai-gateway-infra(de5226d7-33f1-41a5-a919-1c1bfc55962c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" Nov 02 21:00:05.907965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.907944 5800 generic.go:356] "Generic (PLEG): container finished" podID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerID="32a6a571370d331fc27598b5f555efa82695c9a800cec2a95fca2c42c2ca7b80" exitCode=60 Nov 02 21:00:05.908067 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.908024 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" event={"ID":"8b3839a4-e546-4eed-b834-a62cd8b1f41c","Type":"ContainerDied","Data":"32a6a571370d331fc27598b5f555efa82695c9a800cec2a95fca2c42c2ca7b80"} Nov 02 21:00:05.908197 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.908182 5800 scope.go:122] "RemoveContainer" containerID="32a6a571370d331fc27598b5f555efa82695c9a800cec2a95fca2c42c2ca7b80" Nov 02 21:00:05.908443 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.908424 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn_odh-ai-gateway-infra(8b3839a4-e546-4eed-b834-a62cd8b1f41c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" Nov 02 21:00:05.910087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.910058 5800 generic.go:356] "Generic (PLEG): container finished" podID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerID="df61a025fa78b1dc6cf41199bbb0e9e6ad56f2abdf74701d2a553c536ffad6f4" exitCode=60 Nov 02 21:00:05.910181 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.910132 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerDied","Data":"df61a025fa78b1dc6cf41199bbb0e9e6ad56f2abdf74701d2a553c536ffad6f4"} Nov 02 21:00:05.910355 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.910339 5800 scope.go:122] "RemoveContainer" containerID="df61a025fa78b1dc6cf41199bbb0e9e6ad56f2abdf74701d2a553c536ffad6f4" Nov 02 21:00:05.910575 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.910552 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh_odh-ai-gateway-infra(2db514a1-d913-4b14-8e2c-0770313c6d9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" Nov 02 21:00:05.911965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.911949 5800 scope.go:122] "RemoveContainer" containerID="4d8c69f0f0837324b6e41927f7203057dddb33b1a2fc25549696934f149d0012" Nov 02 21:00:05.912182 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.912159 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj_odh-ai-gateway-infra(02fbf823-6929-493a-911c-3404641c8863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" podUID="02fbf823-6929-493a-911c-3404641c8863" Nov 02 21:00:05.913810 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.913788 5800 generic.go:356] "Generic (PLEG): container finished" podID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerID="efbf1b476b5f39af801342891e24b40aca540b92a12097fb64cff41b7b119cba" exitCode=60 Nov 02 21:00:05.913884 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.913839 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerDied","Data":"efbf1b476b5f39af801342891e24b40aca540b92a12097fb64cff41b7b119cba"} Nov 02 21:00:05.914067 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.914053 5800 scope.go:122] "RemoveContainer" containerID="efbf1b476b5f39af801342891e24b40aca540b92a12097fb64cff41b7b119cba" Nov 02 21:00:05.914318 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:05.914297 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc_odh-ai-gateway-infra(ccbf204a-de19-4517-bc00-7bcc299496b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" Nov 02 21:00:05.916106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.916086 5800 scope.go:122] "RemoveContainer" containerID="db8e5603f91a412d3a9f7c55713c401de06833c6c947b1d4e0a9e6616cd22810" Nov 02 21:00:05.931794 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.931771 5800 scope.go:122] "RemoveContainer" containerID="a0b90b4a9b83a4fa43ba2029437b1cb68bd745b35fd10907f68fe6616b69454b" Nov 02 21:00:05.946564 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.946536 5800 scope.go:122] "RemoveContainer" containerID="6cabaf6d81b0f9482e33fe65572f06876c44199291b4a6b3f4d7a7d7d9a1c169" Nov 02 21:00:05.960560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.960535 5800 scope.go:122] "RemoveContainer" containerID="26144ebc75fb3204681fcdf7df030025e220b1702d9ce166f8cab65f328cb9a5" Nov 02 21:00:05.974394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:05.974370 5800 scope.go:122] "RemoveContainer" containerID="08627a3e309c26a7c5bdd9868264c06fdbe4292aa35e4b170aab50bf39453fed" Nov 02 21:00:06.929750 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:06.929726 5800 scope.go:122] "RemoveContainer" containerID="511b1f0d4861dd4ec0dc7fcc2d82c79d6f7ec18ea7f53d2299829507677e0233" Nov 02 21:00:06.930192 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:06.930025 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx_odh-ai-gateway-infra(7fe0e8ef-ce92-40c0-89d4-7c1d91018892)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" Nov 02 21:00:06.931464 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:06.931441 5800 scope.go:122] "RemoveContainer" containerID="074f5588b0ea423309b1820cc799bd32e84d8b0df1356beb40f430a9d306b03a" Nov 02 21:00:06.931728 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:06.931708 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x_odh-ai-gateway-infra(17a63e09-b535-4190-9a7f-f72258817ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" Nov 02 21:00:06.933023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:06.933003 5800 scope.go:122] "RemoveContainer" containerID="32a6a571370d331fc27598b5f555efa82695c9a800cec2a95fca2c42c2ca7b80" Nov 02 21:00:06.933249 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:06.933231 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn_odh-ai-gateway-infra(8b3839a4-e546-4eed-b834-a62cd8b1f41c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" Nov 02 21:00:06.934462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:06.934446 5800 scope.go:122] "RemoveContainer" containerID="df61a025fa78b1dc6cf41199bbb0e9e6ad56f2abdf74701d2a553c536ffad6f4" Nov 02 21:00:06.934686 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:06.934669 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh_odh-ai-gateway-infra(2db514a1-d913-4b14-8e2c-0770313c6d9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" Nov 02 21:00:06.935932 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:06.935915 5800 scope.go:122] "RemoveContainer" containerID="efbf1b476b5f39af801342891e24b40aca540b92a12097fb64cff41b7b119cba" Nov 02 21:00:06.936172 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:06.936147 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc_odh-ai-gateway-infra(ccbf204a-de19-4517-bc00-7bcc299496b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" Nov 02 21:00:06.937660 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:06.937643 5800 scope.go:122] "RemoveContainer" containerID="8f2b9e97acb1c0b41e6d50c73903bc3e842dee69df3c6aec6d3e6eb589d75202" Nov 02 21:00:06.937922 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:06.937878 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj_odh-ai-gateway-infra(33550673-a1d4-4cb5-9467-a20cc86f3823)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" Nov 02 21:00:06.939236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:06.939218 5800 scope.go:122] "RemoveContainer" containerID="05e88ab3a6120c1ccb0a47e85b354bf5908141b76d7aff06d7f3597f4ce5e41d" Nov 02 21:00:06.939478 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:06.939458 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8_odh-ai-gateway-infra(5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" Nov 02 21:00:15.615434 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:15.615398 5800 scope.go:122] "RemoveContainer" containerID="a35cf1d346831c9a793a408c3406b33ab8adf9a4b9b283c62843c1c28a67c801" Nov 02 21:00:16.615140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:16.615111 5800 scope.go:122] "RemoveContainer" containerID="4d8c69f0f0837324b6e41927f7203057dddb33b1a2fc25549696934f149d0012" Nov 02 21:00:16.615354 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:16.615303 5800 scope.go:122] "RemoveContainer" containerID="9125f61cedc93f4162d040ee04fef1189fb39b1d4e95a67659fa95945f5a34ec" Nov 02 21:00:17.022484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:17.022452 5800 generic.go:356] "Generic (PLEG): container finished" podID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerID="7f935a741cd3a0da2c3cff31892e13f1585f61fbcf419c428cd3cd9114e1d6ab" exitCode=60 Nov 02 21:00:17.022974 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:17.022524 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" event={"ID":"b37f3722-ebfe-4975-8b8f-6cd72a3c473e","Type":"ContainerDied","Data":"7f935a741cd3a0da2c3cff31892e13f1585f61fbcf419c428cd3cd9114e1d6ab"} Nov 02 21:00:17.022974 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:17.022583 5800 scope.go:122] "RemoveContainer" containerID="a35cf1d346831c9a793a408c3406b33ab8adf9a4b9b283c62843c1c28a67c801" Nov 02 21:00:17.023106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:17.023051 5800 scope.go:122] "RemoveContainer" containerID="7f935a741cd3a0da2c3cff31892e13f1585f61fbcf419c428cd3cd9114e1d6ab" Nov 02 21:00:17.023431 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:17.023408 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf_odh-ai-gateway-infra(b37f3722-ebfe-4975-8b8f-6cd72a3c473e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" Nov 02 21:00:17.615339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:17.615309 5800 scope.go:122] "RemoveContainer" containerID="dff908de3796e2d92b32113f93d86d94c2b4911aeab9e5d9330e6574d2f5091d" Nov 02 21:00:17.615531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:17.615384 5800 scope.go:122] "RemoveContainer" containerID="32a6a571370d331fc27598b5f555efa82695c9a800cec2a95fca2c42c2ca7b80" Nov 02 21:00:18.036232 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.036204 5800 generic.go:356] "Generic (PLEG): container finished" podID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerID="618ec6a4bf6888ca124c5a69f040cbb825126c83c4bf5c9ee54520867e233bc1" exitCode=60 Nov 02 21:00:18.036693 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.036271 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" event={"ID":"4baedbb3-e6fa-4a67-8e7f-a27005a97712","Type":"ContainerDied","Data":"618ec6a4bf6888ca124c5a69f040cbb825126c83c4bf5c9ee54520867e233bc1"} Nov 02 21:00:18.036693 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.036354 5800 scope.go:122] "RemoveContainer" containerID="9125f61cedc93f4162d040ee04fef1189fb39b1d4e95a67659fa95945f5a34ec" Nov 02 21:00:18.036876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.036785 5800 scope.go:122] "RemoveContainer" containerID="618ec6a4bf6888ca124c5a69f040cbb825126c83c4bf5c9ee54520867e233bc1" Nov 02 21:00:18.037200 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:18.037170 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk_odh-ai-gateway-infra(4baedbb3-e6fa-4a67-8e7f-a27005a97712)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" Nov 02 21:00:18.040958 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.040932 5800 generic.go:356] "Generic (PLEG): container finished" podID="02fbf823-6929-493a-911c-3404641c8863" containerID="6dcabe8ffb96741a714c9459c7c6abd33ea1d0c6b6140d1b609689542a317a3f" exitCode=60 Nov 02 21:00:18.041078 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.040978 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" event={"ID":"02fbf823-6929-493a-911c-3404641c8863","Type":"ContainerDied","Data":"6dcabe8ffb96741a714c9459c7c6abd33ea1d0c6b6140d1b609689542a317a3f"} Nov 02 21:00:18.041532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.041514 5800 scope.go:122] "RemoveContainer" containerID="6dcabe8ffb96741a714c9459c7c6abd33ea1d0c6b6140d1b609689542a317a3f" Nov 02 21:00:18.041891 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:18.041865 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj_odh-ai-gateway-infra(02fbf823-6929-493a-911c-3404641c8863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" podUID="02fbf823-6929-493a-911c-3404641c8863" Nov 02 21:00:18.053049 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.053003 5800 scope.go:122] "RemoveContainer" containerID="4d8c69f0f0837324b6e41927f7203057dddb33b1a2fc25549696934f149d0012" Nov 02 21:00:18.141200 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.141172 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf"] Nov 02 21:00:18.319925 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.319898 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:18.483759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.483664 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-maas-api-service-ca\") pod \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " Nov 02 21:00:18.483960 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.483782 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-kube-api-access-9rvn5\" (UniqueName: \"kubernetes.io/projected/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-kube-api-access-9rvn5\") pod \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\" (UID: \"b37f3722-ebfe-4975-8b8f-6cd72a3c473e\") " Nov 02 21:00:18.484128 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.484104 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-maas-api-service-ca" pod "b37f3722-ebfe-4975-8b8f-6cd72a3c473e" (UID: "b37f3722-ebfe-4975-8b8f-6cd72a3c473e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:18.484400 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.484381 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:18.486192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.486161 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-kube-api-access-9rvn5" pod "b37f3722-ebfe-4975-8b8f-6cd72a3c473e" (UID: "b37f3722-ebfe-4975-8b8f-6cd72a3c473e"). InnerVolumeSpecName "kube-api-access-9rvn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:18.585692 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.585646 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvn5\" (UniqueName: \"kubernetes.io/projected/b37f3722-ebfe-4975-8b8f-6cd72a3c473e-kube-api-access-9rvn5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:18.614794 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.614626 5800 scope.go:122] "RemoveContainer" containerID="1125ae4b6ceb468f5fcea077d89907c89ed112d9795ca2961a086b013485cb4a" Nov 02 21:00:18.614794 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.614784 5800 scope.go:122] "RemoveContainer" containerID="8f2b9e97acb1c0b41e6d50c73903bc3e842dee69df3c6aec6d3e6eb589d75202" Nov 02 21:00:18.615113 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.615092 5800 scope.go:122] "RemoveContainer" containerID="df61a025fa78b1dc6cf41199bbb0e9e6ad56f2abdf74701d2a553c536ffad6f4" Nov 02 21:00:18.615487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.615430 5800 scope.go:122] "RemoveContainer" containerID="1dc2569fdb9313eb50e30ec685de41f6f3e03e15f04df18da1695c742e1d1cb8" Nov 02 21:00:18.615763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.615739 5800 scope.go:122] "RemoveContainer" containerID="efbf1b476b5f39af801342891e24b40aca540b92a12097fb64cff41b7b119cba" Nov 02 21:00:18.615909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.615810 5800 scope.go:122] "RemoveContainer" containerID="074f5588b0ea423309b1820cc799bd32e84d8b0df1356beb40f430a9d306b03a" Nov 02 21:00:18.616051 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:18.616031 5800 scope.go:122] "RemoveContainer" containerID="05e88ab3a6120c1ccb0a47e85b354bf5908141b76d7aff06d7f3597f4ce5e41d" Nov 02 21:00:19.056694 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.056661 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" event={"ID":"b37f3722-ebfe-4975-8b8f-6cd72a3c473e","Type":"ContainerDied","Data":"dc366ceb3b789b14cd13204c4674f1ce0fa4b8428523dc343c220ed113827776"} Nov 02 21:00:19.057211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.056717 5800 scope.go:122] "RemoveContainer" containerID="7f935a741cd3a0da2c3cff31892e13f1585f61fbcf419c428cd3cd9114e1d6ab" Nov 02 21:00:19.057211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.056803 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf" Nov 02 21:00:19.060722 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.060692 5800 generic.go:356] "Generic (PLEG): container finished" podID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerID="a764d8c6633dbab4297e4dba6cff4156cac79ee40f2db28a16fe0b4fb42a1318" exitCode=60 Nov 02 21:00:19.060853 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.060775 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" event={"ID":"8b3839a4-e546-4eed-b834-a62cd8b1f41c","Type":"ContainerDied","Data":"a764d8c6633dbab4297e4dba6cff4156cac79ee40f2db28a16fe0b4fb42a1318"} Nov 02 21:00:19.061241 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.061221 5800 scope.go:122] "RemoveContainer" containerID="a764d8c6633dbab4297e4dba6cff4156cac79ee40f2db28a16fe0b4fb42a1318" Nov 02 21:00:19.061714 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:19.061688 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn_odh-ai-gateway-infra(8b3839a4-e546-4eed-b834-a62cd8b1f41c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" Nov 02 21:00:19.066835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.066587 5800 generic.go:356] "Generic (PLEG): container finished" podID="529e13a2-3e07-4af5-aaea-c3942a800589" containerID="f738d84a937611a037ff52ef09aecb3835b99302004210f7293f77347a102c59" exitCode=60 Nov 02 21:00:19.066835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.066628 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" event={"ID":"529e13a2-3e07-4af5-aaea-c3942a800589","Type":"ContainerDied","Data":"f738d84a937611a037ff52ef09aecb3835b99302004210f7293f77347a102c59"} Nov 02 21:00:19.067598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.067276 5800 scope.go:122] "RemoveContainer" containerID="f738d84a937611a037ff52ef09aecb3835b99302004210f7293f77347a102c59" Nov 02 21:00:19.067598 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:19.067557 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k_odh-ai-gateway-infra(529e13a2-3e07-4af5-aaea-c3942a800589)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" Nov 02 21:00:19.073272 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.073250 5800 scope.go:122] "RemoveContainer" containerID="32a6a571370d331fc27598b5f555efa82695c9a800cec2a95fca2c42c2ca7b80" Nov 02 21:00:19.088701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.088675 5800 scope.go:122] "RemoveContainer" containerID="dff908de3796e2d92b32113f93d86d94c2b4911aeab9e5d9330e6574d2f5091d" Nov 02 21:00:19.159572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.159543 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk"] Nov 02 21:00:19.214658 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.214615 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf"] Nov 02 21:00:19.220882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.220850 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2c6ce9-29894220-fwqwf"] Nov 02 21:00:19.320961 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.320935 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:19.372383 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.372297 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj"] Nov 02 21:00:19.496813 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.496780 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4baedbb3-e6fa-4a67-8e7f-a27005a97712-kube-api-access-snptt\" (UniqueName: \"kubernetes.io/projected/4baedbb3-e6fa-4a67-8e7f-a27005a97712-kube-api-access-snptt\") pod \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " Nov 02 21:00:19.496992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.496833 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4baedbb3-e6fa-4a67-8e7f-a27005a97712-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baedbb3-e6fa-4a67-8e7f-a27005a97712-maas-api-service-ca\") pod \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\" (UID: \"4baedbb3-e6fa-4a67-8e7f-a27005a97712\") " Nov 02 21:00:19.497211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.497185 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4baedbb3-e6fa-4a67-8e7f-a27005a97712-maas-api-service-ca" pod "4baedbb3-e6fa-4a67-8e7f-a27005a97712" (UID: "4baedbb3-e6fa-4a67-8e7f-a27005a97712"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:19.497543 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.497508 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4baedbb3-e6fa-4a67-8e7f-a27005a97712-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:19.532836 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.532757 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4baedbb3-e6fa-4a67-8e7f-a27005a97712-kube-api-access-snptt" pod "4baedbb3-e6fa-4a67-8e7f-a27005a97712" (UID: "4baedbb3-e6fa-4a67-8e7f-a27005a97712"). InnerVolumeSpecName "kube-api-access-snptt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:19.598764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.598730 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snptt\" (UniqueName: \"kubernetes.io/projected/4baedbb3-e6fa-4a67-8e7f-a27005a97712-kube-api-access-snptt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:19.622052 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.621464 5800 scope.go:122] "RemoveContainer" containerID="511b1f0d4861dd4ec0dc7fcc2d82c79d6f7ec18ea7f53d2299829507677e0233" Nov 02 21:00:19.651780 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.651709 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" path="/var/lib/kubelet/pods/b37f3722-ebfe-4975-8b8f-6cd72a3c473e/volumes" Nov 02 21:00:19.836903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:19.836876 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:20.003372 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.003337 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02fbf823-6929-493a-911c-3404641c8863-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fbf823-6929-493a-911c-3404641c8863-maas-api-service-ca\") pod \"02fbf823-6929-493a-911c-3404641c8863\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " Nov 02 21:00:20.003560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.003535 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fbf823-6929-493a-911c-3404641c8863-kube-api-access-jvc2v\" (UniqueName: \"kubernetes.io/projected/02fbf823-6929-493a-911c-3404641c8863-kube-api-access-jvc2v\") pod \"02fbf823-6929-493a-911c-3404641c8863\" (UID: \"02fbf823-6929-493a-911c-3404641c8863\") " Nov 02 21:00:20.003716 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.003693 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fbf823-6929-493a-911c-3404641c8863-maas-api-service-ca" pod "02fbf823-6929-493a-911c-3404641c8863" (UID: "02fbf823-6929-493a-911c-3404641c8863"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:20.003941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.003927 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02fbf823-6929-493a-911c-3404641c8863-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:20.005775 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.005749 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fbf823-6929-493a-911c-3404641c8863-kube-api-access-jvc2v" pod "02fbf823-6929-493a-911c-3404641c8863" (UID: "02fbf823-6929-493a-911c-3404641c8863"). InnerVolumeSpecName "kube-api-access-jvc2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:20.077575 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.077543 5800 generic.go:356] "Generic (PLEG): container finished" podID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerID="1756ec0b03dc7fbe00e439a2bdace29999ac09b4b199bd10044a97d6857301f9" exitCode=60 Nov 02 21:00:20.078006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.077614 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" event={"ID":"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939","Type":"ContainerDied","Data":"1756ec0b03dc7fbe00e439a2bdace29999ac09b4b199bd10044a97d6857301f9"} Nov 02 21:00:20.078006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.077674 5800 scope.go:122] "RemoveContainer" containerID="05e88ab3a6120c1ccb0a47e85b354bf5908141b76d7aff06d7f3597f4ce5e41d" Nov 02 21:00:20.078156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.078107 5800 scope.go:122] "RemoveContainer" containerID="1756ec0b03dc7fbe00e439a2bdace29999ac09b4b199bd10044a97d6857301f9" Nov 02 21:00:20.078519 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:20.078494 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8_odh-ai-gateway-infra(5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" Nov 02 21:00:20.080602 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.080584 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" Nov 02 21:00:20.080731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.080627 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk" event={"ID":"4baedbb3-e6fa-4a67-8e7f-a27005a97712","Type":"ContainerDied","Data":"42e20dbabe0303021b73d5ad27397e4179817939de2426b8ac37d48409fa36e6"} Nov 02 21:00:20.082543 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.082509 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerStarted","Data":"73c85462402b11573cd1a129cb83446991b3539328c0ab95e0ca7a52cc0ff847"} Nov 02 21:00:20.085464 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.085443 5800 generic.go:356] "Generic (PLEG): container finished" podID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerID="6db23322f50ea2917c7533c11fa9b0d0df449d5108b51e8bb6db454ded748b39" exitCode=60 Nov 02 21:00:20.085573 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.085511 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" event={"ID":"de5226d7-33f1-41a5-a919-1c1bfc55962c","Type":"ContainerDied","Data":"6db23322f50ea2917c7533c11fa9b0d0df449d5108b51e8bb6db454ded748b39"} Nov 02 21:00:20.085873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.085859 5800 scope.go:122] "RemoveContainer" containerID="6db23322f50ea2917c7533c11fa9b0d0df449d5108b51e8bb6db454ded748b39" Nov 02 21:00:20.086142 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:20.086124 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc_odh-ai-gateway-infra(de5226d7-33f1-41a5-a919-1c1bfc55962c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" Nov 02 21:00:20.088818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.088799 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerStarted","Data":"d091835a239025220f0ddcda4598c95f3b17743244ce75478a5b1d362141ab64"} Nov 02 21:00:20.093826 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.092803 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" event={"ID":"02fbf823-6929-493a-911c-3404641c8863","Type":"ContainerDied","Data":"5ed788eb3aef952abbab9849b81b25afe40b0e3190d8a6f4d72ac4685e4b0d85"} Nov 02 21:00:20.093826 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.092890 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj" Nov 02 21:00:20.096739 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.096716 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerStarted","Data":"be86376d969b35fbcab466710ce1c602c9d0fbc7d7e49f8064bbe5beb323d328"} Nov 02 21:00:20.098749 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.098728 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerStarted","Data":"b9831aa1a1d0ea20445f10aa6041659cb62b388c64a8362a5ee45edefe7f4950"} Nov 02 21:00:20.100599 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.100580 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" event={"ID":"2f4090b1-df6d-4014-8524-f39154343d5b","Type":"ContainerStarted","Data":"50e74ad89f6228b29538cfbf78893accb5079f796f2d3a79a4f91b4d16205cfb"} Nov 02 21:00:20.105484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.105464 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvc2v\" (UniqueName: \"kubernetes.io/projected/02fbf823-6929-493a-911c-3404641c8863-kube-api-access-jvc2v\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:20.337054 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.337016 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k"] Nov 02 21:00:20.430631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.430526 5800 scope.go:122] "RemoveContainer" containerID="618ec6a4bf6888ca124c5a69f040cbb825126c83c4bf5c9ee54520867e233bc1" Nov 02 21:00:20.442348 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.441326 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn"] Nov 02 21:00:20.450554 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.450459 5800 scope.go:122] "RemoveContainer" containerID="1125ae4b6ceb468f5fcea077d89907c89ed112d9795ca2961a086b013485cb4a" Nov 02 21:00:20.478613 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.478580 5800 scope.go:122] "RemoveContainer" containerID="6dcabe8ffb96741a714c9459c7c6abd33ea1d0c6b6140d1b609689542a317a3f" Nov 02 21:00:20.517112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.517088 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:20.614373 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.614332 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529e13a2-3e07-4af5-aaea-c3942a800589-kube-api-access-4ph6r\" (UniqueName: \"kubernetes.io/projected/529e13a2-3e07-4af5-aaea-c3942a800589-kube-api-access-4ph6r\") pod \"529e13a2-3e07-4af5-aaea-c3942a800589\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " Nov 02 21:00:20.614930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.614904 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529e13a2-3e07-4af5-aaea-c3942a800589-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529e13a2-3e07-4af5-aaea-c3942a800589-maas-api-service-ca\") pod \"529e13a2-3e07-4af5-aaea-c3942a800589\" (UID: \"529e13a2-3e07-4af5-aaea-c3942a800589\") " Nov 02 21:00:20.617261 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.617140 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529e13a2-3e07-4af5-aaea-c3942a800589-maas-api-service-ca" pod "529e13a2-3e07-4af5-aaea-c3942a800589" (UID: "529e13a2-3e07-4af5-aaea-c3942a800589"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:20.617757 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.617725 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/529e13a2-3e07-4af5-aaea-c3942a800589-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:20.618265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.618232 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529e13a2-3e07-4af5-aaea-c3942a800589-kube-api-access-4ph6r" pod "529e13a2-3e07-4af5-aaea-c3942a800589" (UID: "529e13a2-3e07-4af5-aaea-c3942a800589"). InnerVolumeSpecName "kube-api-access-4ph6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:20.649714 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.649683 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:20.718917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.718881 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3839a4-e546-4eed-b834-a62cd8b1f41c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3839a4-e546-4eed-b834-a62cd8b1f41c-maas-api-service-ca\") pod \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " Nov 02 21:00:20.719096 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.719065 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3839a4-e546-4eed-b834-a62cd8b1f41c-kube-api-access-ccp2z\" (UniqueName: \"kubernetes.io/projected/8b3839a4-e546-4eed-b834-a62cd8b1f41c-kube-api-access-ccp2z\") pod \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\" (UID: \"8b3839a4-e546-4eed-b834-a62cd8b1f41c\") " Nov 02 21:00:20.719259 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.719235 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3839a4-e546-4eed-b834-a62cd8b1f41c-maas-api-service-ca" pod "8b3839a4-e546-4eed-b834-a62cd8b1f41c" (UID: "8b3839a4-e546-4eed-b834-a62cd8b1f41c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:20.719535 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.719516 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3839a4-e546-4eed-b834-a62cd8b1f41c-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:20.719607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.719538 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ph6r\" (UniqueName: \"kubernetes.io/projected/529e13a2-3e07-4af5-aaea-c3942a800589-kube-api-access-4ph6r\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:20.721298 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.721260 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3839a4-e546-4eed-b834-a62cd8b1f41c-kube-api-access-ccp2z" pod "8b3839a4-e546-4eed-b834-a62cd8b1f41c" (UID: "8b3839a4-e546-4eed-b834-a62cd8b1f41c"). InnerVolumeSpecName "kube-api-access-ccp2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:20.820840 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:20.820801 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccp2z\" (UniqueName: \"kubernetes.io/projected/8b3839a4-e546-4eed-b834-a62cd8b1f41c-kube-api-access-ccp2z\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:21.113734 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.113699 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" Nov 02 21:00:21.113734 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.113726 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k" event={"ID":"529e13a2-3e07-4af5-aaea-c3942a800589","Type":"ContainerDied","Data":"bf900659da238b4c8bb58b9837e05ba47208ce2a3a5815d529c35ab67c5313d6"} Nov 02 21:00:21.114431 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.113798 5800 scope.go:122] "RemoveContainer" containerID="f738d84a937611a037ff52ef09aecb3835b99302004210f7293f77347a102c59" Nov 02 21:00:21.118403 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.118376 5800 generic.go:356] "Generic (PLEG): container finished" podID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerID="67d946516e3c4c249c907109fb294347c562385450ae29279e020bae33d51c47" exitCode=60 Nov 02 21:00:21.118525 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.118450 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" event={"ID":"7fe0e8ef-ce92-40c0-89d4-7c1d91018892","Type":"ContainerDied","Data":"67d946516e3c4c249c907109fb294347c562385450ae29279e020bae33d51c47"} Nov 02 21:00:21.118961 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.118940 5800 scope.go:122] "RemoveContainer" containerID="67d946516e3c4c249c907109fb294347c562385450ae29279e020bae33d51c47" Nov 02 21:00:21.119512 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:21.119477 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx_odh-ai-gateway-infra(7fe0e8ef-ce92-40c0-89d4-7c1d91018892)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" Nov 02 21:00:21.120743 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.120724 5800 generic.go:356] "Generic (PLEG): container finished" podID="17a63e09-b535-4190-9a7f-f72258817ebc" containerID="73c85462402b11573cd1a129cb83446991b3539328c0ab95e0ca7a52cc0ff847" exitCode=60 Nov 02 21:00:21.120911 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.120894 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerDied","Data":"73c85462402b11573cd1a129cb83446991b3539328c0ab95e0ca7a52cc0ff847"} Nov 02 21:00:21.121226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.121209 5800 scope.go:122] "RemoveContainer" containerID="73c85462402b11573cd1a129cb83446991b3539328c0ab95e0ca7a52cc0ff847" Nov 02 21:00:21.121480 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:21.121458 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x_odh-ai-gateway-infra(17a63e09-b535-4190-9a7f-f72258817ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" Nov 02 21:00:21.123937 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.123917 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" event={"ID":"8b3839a4-e546-4eed-b834-a62cd8b1f41c","Type":"ContainerDied","Data":"229c3d4118e855f54a1eb581b9cab51567cb4b7e363a7baf4ec2830fd97c12e9"} Nov 02 21:00:21.124029 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.123988 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn" Nov 02 21:00:21.126872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.126345 5800 generic.go:356] "Generic (PLEG): container finished" podID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerID="d091835a239025220f0ddcda4598c95f3b17743244ce75478a5b1d362141ab64" exitCode=60 Nov 02 21:00:21.126872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.126418 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerDied","Data":"d091835a239025220f0ddcda4598c95f3b17743244ce75478a5b1d362141ab64"} Nov 02 21:00:21.126872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.126770 5800 scope.go:122] "RemoveContainer" containerID="d091835a239025220f0ddcda4598c95f3b17743244ce75478a5b1d362141ab64" Nov 02 21:00:21.127530 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:21.127506 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh_odh-ai-gateway-infra(2db514a1-d913-4b14-8e2c-0770313c6d9d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" Nov 02 21:00:21.127874 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.127860 5800 scope.go:122] "RemoveContainer" containerID="511b1f0d4861dd4ec0dc7fcc2d82c79d6f7ec18ea7f53d2299829507677e0233" Nov 02 21:00:21.129910 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.129720 5800 generic.go:356] "Generic (PLEG): container finished" podID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerID="be86376d969b35fbcab466710ce1c602c9d0fbc7d7e49f8064bbe5beb323d328" exitCode=60 Nov 02 21:00:21.129910 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.129773 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerDied","Data":"be86376d969b35fbcab466710ce1c602c9d0fbc7d7e49f8064bbe5beb323d328"} Nov 02 21:00:21.130146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.130125 5800 scope.go:122] "RemoveContainer" containerID="be86376d969b35fbcab466710ce1c602c9d0fbc7d7e49f8064bbe5beb323d328" Nov 02 21:00:21.130351 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:21.130334 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc_odh-ai-gateway-infra(ccbf204a-de19-4517-bc00-7bcc299496b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" Nov 02 21:00:21.132693 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.132668 5800 generic.go:356] "Generic (PLEG): container finished" podID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerID="b9831aa1a1d0ea20445f10aa6041659cb62b388c64a8362a5ee45edefe7f4950" exitCode=60 Nov 02 21:00:21.132816 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.132752 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerDied","Data":"b9831aa1a1d0ea20445f10aa6041659cb62b388c64a8362a5ee45edefe7f4950"} Nov 02 21:00:21.133104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.133073 5800 scope.go:122] "RemoveContainer" containerID="b9831aa1a1d0ea20445f10aa6041659cb62b388c64a8362a5ee45edefe7f4950" Nov 02 21:00:21.133363 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:21.133317 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj_odh-ai-gateway-infra(33550673-a1d4-4cb5-9467-a20cc86f3823)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" Nov 02 21:00:21.135310 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.135271 5800 generic.go:356] "Generic (PLEG): container finished" podID="2f4090b1-df6d-4014-8524-f39154343d5b" containerID="50e74ad89f6228b29538cfbf78893accb5079f796f2d3a79a4f91b4d16205cfb" exitCode=60 Nov 02 21:00:21.135455 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.135312 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" event={"ID":"2f4090b1-df6d-4014-8524-f39154343d5b","Type":"ContainerDied","Data":"50e74ad89f6228b29538cfbf78893accb5079f796f2d3a79a4f91b4d16205cfb"} Nov 02 21:00:21.135595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.135582 5800 scope.go:122] "RemoveContainer" containerID="50e74ad89f6228b29538cfbf78893accb5079f796f2d3a79a4f91b4d16205cfb" Nov 02 21:00:21.135766 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:21.135749 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv_odh-ai-gateway-infra(2f4090b1-df6d-4014-8524-f39154343d5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" Nov 02 21:00:21.152870 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.152843 5800 scope.go:122] "RemoveContainer" containerID="074f5588b0ea423309b1820cc799bd32e84d8b0df1356beb40f430a9d306b03a" Nov 02 21:00:21.167673 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.167652 5800 scope.go:122] "RemoveContainer" containerID="a764d8c6633dbab4297e4dba6cff4156cac79ee40f2db28a16fe0b4fb42a1318" Nov 02 21:00:21.181145 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.181088 5800 scope.go:122] "RemoveContainer" containerID="df61a025fa78b1dc6cf41199bbb0e9e6ad56f2abdf74701d2a553c536ffad6f4" Nov 02 21:00:21.198505 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.198475 5800 scope.go:122] "RemoveContainer" containerID="efbf1b476b5f39af801342891e24b40aca540b92a12097fb64cff41b7b119cba" Nov 02 21:00:21.208323 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.204429 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8"] Nov 02 21:00:21.224242 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.224215 5800 scope.go:122] "RemoveContainer" containerID="8f2b9e97acb1c0b41e6d50c73903bc3e842dee69df3c6aec6d3e6eb589d75202" Nov 02 21:00:21.258570 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.258490 5800 scope.go:122] "RemoveContainer" containerID="1dc2569fdb9313eb50e30ec685de41f6f3e03e15f04df18da1695c742e1d1cb8" Nov 02 21:00:21.281035 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.280994 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk"] Nov 02 21:00:21.285335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.285308 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-44a484-29894220-5tlvk"] Nov 02 21:00:21.351529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.351491 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj"] Nov 02 21:00:21.382385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.382351 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj"] Nov 02 21:00:21.386207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.386180 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894220-crpnj"] Nov 02 21:00:21.393751 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.393729 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:21.458101 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.458060 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh"] Nov 02 21:00:21.487541 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.487505 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn"] Nov 02 21:00:21.489947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.489917 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-maas-api-service-ca\") pod \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " Nov 02 21:00:21.490187 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.490059 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-kube-api-access-fkqmh\" (UniqueName: \"kubernetes.io/projected/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-kube-api-access-fkqmh\") pod \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\" (UID: \"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939\") " Nov 02 21:00:21.490373 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.490350 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-maas-api-service-ca" pod "5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" (UID: "5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:21.490595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.490581 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:21.491586 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.491565 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-578636-29894220-lv7zn"] Nov 02 21:00:21.492189 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.492166 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-kube-api-access-fkqmh" pod "5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" (UID: "5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939"). InnerVolumeSpecName "kube-api-access-fkqmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:21.566245 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.566195 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc"] Nov 02 21:00:21.591946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.591902 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkqmh\" (UniqueName: \"kubernetes.io/projected/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939-kube-api-access-fkqmh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:21.628568 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.628491 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02fbf823-6929-493a-911c-3404641c8863" path="/var/lib/kubelet/pods/02fbf823-6929-493a-911c-3404641c8863/volumes" Nov 02 21:00:21.628966 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.628952 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" path="/var/lib/kubelet/pods/4baedbb3-e6fa-4a67-8e7f-a27005a97712/volumes" Nov 02 21:00:21.629390 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.629375 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" path="/var/lib/kubelet/pods/8b3839a4-e546-4eed-b834-a62cd8b1f41c/volumes" Nov 02 21:00:21.741951 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.741917 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x"] Nov 02 21:00:21.848387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.848353 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv"] Nov 02 21:00:21.952863 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:21.952775 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc"] Nov 02 21:00:22.152270 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.152234 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" event={"ID":"de5226d7-33f1-41a5-a919-1c1bfc55962c","Type":"ContainerDied","Data":"cc4d8588f5465e9cda31f80d9b43eaa5697ccc1e1a2d0eeec6fd0053f999ef28"} Nov 02 21:00:22.152760 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.152327 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc4d8588f5465e9cda31f80d9b43eaa5697ccc1e1a2d0eeec6fd0053f999ef28" Nov 02 21:00:22.154945 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.154921 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:22.166741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.165818 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" event={"ID":"5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939","Type":"ContainerDied","Data":"b485c7d4205309e7735051449fcba3a7ef1dc982c660fc0a48944d9e2fd95a9a"} Nov 02 21:00:22.166741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.165890 5800 scope.go:122] "RemoveContainer" containerID="1756ec0b03dc7fbe00e439a2bdace29999ac09b4b199bd10044a97d6857301f9" Nov 02 21:00:22.166741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.166026 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8" Nov 02 21:00:22.302362 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.301749 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5226d7-33f1-41a5-a919-1c1bfc55962c-kube-api-access-f65wg\" (UniqueName: \"kubernetes.io/projected/de5226d7-33f1-41a5-a919-1c1bfc55962c-kube-api-access-f65wg\") pod \"de5226d7-33f1-41a5-a919-1c1bfc55962c\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " Nov 02 21:00:22.302362 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.302014 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5226d7-33f1-41a5-a919-1c1bfc55962c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5226d7-33f1-41a5-a919-1c1bfc55962c-maas-api-service-ca\") pod \"de5226d7-33f1-41a5-a919-1c1bfc55962c\" (UID: \"de5226d7-33f1-41a5-a919-1c1bfc55962c\") " Nov 02 21:00:22.303027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.302609 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5226d7-33f1-41a5-a919-1c1bfc55962c-maas-api-service-ca" pod "de5226d7-33f1-41a5-a919-1c1bfc55962c" (UID: "de5226d7-33f1-41a5-a919-1c1bfc55962c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:22.303027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.302989 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5226d7-33f1-41a5-a919-1c1bfc55962c-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:22.343869 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.339228 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5226d7-33f1-41a5-a919-1c1bfc55962c-kube-api-access-f65wg" pod "de5226d7-33f1-41a5-a919-1c1bfc55962c" (UID: "de5226d7-33f1-41a5-a919-1c1bfc55962c"). InnerVolumeSpecName "kube-api-access-f65wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:22.369483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.355919 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k"] Nov 02 21:00:22.369483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.363123 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-426941-29894220-4sq8k"] Nov 02 21:00:22.403952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.403914 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f65wg\" (UniqueName: \"kubernetes.io/projected/de5226d7-33f1-41a5-a919-1c1bfc55962c-kube-api-access-f65wg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:22.444585 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.444554 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:22.629457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.619859 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbf204a-de19-4517-bc00-7bcc299496b3-kube-api-access-tsbdf\" (UniqueName: \"kubernetes.io/projected/ccbf204a-de19-4517-bc00-7bcc299496b3-kube-api-access-tsbdf\") pod \"ccbf204a-de19-4517-bc00-7bcc299496b3\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " Nov 02 21:00:22.629457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.619916 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbf204a-de19-4517-bc00-7bcc299496b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf204a-de19-4517-bc00-7bcc299496b3-maas-api-service-ca\") pod \"ccbf204a-de19-4517-bc00-7bcc299496b3\" (UID: \"ccbf204a-de19-4517-bc00-7bcc299496b3\") " Nov 02 21:00:22.631076 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.630843 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbf204a-de19-4517-bc00-7bcc299496b3-maas-api-service-ca" pod "ccbf204a-de19-4517-bc00-7bcc299496b3" (UID: "ccbf204a-de19-4517-bc00-7bcc299496b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:22.748459 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.748342 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbf204a-de19-4517-bc00-7bcc299496b3-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:22.901196 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.890642 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx"] Nov 02 21:00:22.944074 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:22.944029 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:00:22.947186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.947156 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:22.958433 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.958348 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:22.968548 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.968460 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:22.972735 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.972697 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbf204a-de19-4517-bc00-7bcc299496b3-kube-api-access-tsbdf" pod "ccbf204a-de19-4517-bc00-7bcc299496b3" (UID: "ccbf204a-de19-4517-bc00-7bcc299496b3"). InnerVolumeSpecName "kube-api-access-tsbdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:22.974545 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:22.974463 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:23.063554 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.063310 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8"] Nov 02 21:00:23.070885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.070816 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a9ed60-29894220-9k5t8"] Nov 02 21:00:23.073622 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.073357 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a63e09-b535-4190-9a7f-f72258817ebc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a63e09-b535-4190-9a7f-f72258817ebc-maas-api-service-ca\") pod \"17a63e09-b535-4190-9a7f-f72258817ebc\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " Nov 02 21:00:23.074031 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.074007 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33550673-a1d4-4cb5-9467-a20cc86f3823-kube-api-access-5bwr8\" (UniqueName: \"kubernetes.io/projected/33550673-a1d4-4cb5-9467-a20cc86f3823-kube-api-access-5bwr8\") pod \"33550673-a1d4-4cb5-9467-a20cc86f3823\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " Nov 02 21:00:23.074587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.074567 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4090b1-df6d-4014-8524-f39154343d5b-kube-api-access-ql5gs\" (UniqueName: \"kubernetes.io/projected/2f4090b1-df6d-4014-8524-f39154343d5b-kube-api-access-ql5gs\") pod \"2f4090b1-df6d-4014-8524-f39154343d5b\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " Nov 02 21:00:23.075082 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.075062 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33550673-a1d4-4cb5-9467-a20cc86f3823-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33550673-a1d4-4cb5-9467-a20cc86f3823-maas-api-service-ca\") pod \"33550673-a1d4-4cb5-9467-a20cc86f3823\" (UID: \"33550673-a1d4-4cb5-9467-a20cc86f3823\") " Nov 02 21:00:23.075708 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.075688 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db514a1-d913-4b14-8e2c-0770313c6d9d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db514a1-d913-4b14-8e2c-0770313c6d9d-maas-api-service-ca\") pod \"2db514a1-d913-4b14-8e2c-0770313c6d9d\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " Nov 02 21:00:23.076257 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.076237 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a63e09-b535-4190-9a7f-f72258817ebc-kube-api-access-5scpf\" (UniqueName: \"kubernetes.io/projected/17a63e09-b535-4190-9a7f-f72258817ebc-kube-api-access-5scpf\") pod \"17a63e09-b535-4190-9a7f-f72258817ebc\" (UID: \"17a63e09-b535-4190-9a7f-f72258817ebc\") " Nov 02 21:00:23.076763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.076744 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db514a1-d913-4b14-8e2c-0770313c6d9d-kube-api-access-6bqfg\" (UniqueName: \"kubernetes.io/projected/2db514a1-d913-4b14-8e2c-0770313c6d9d-kube-api-access-6bqfg\") pod \"2db514a1-d913-4b14-8e2c-0770313c6d9d\" (UID: \"2db514a1-d913-4b14-8e2c-0770313c6d9d\") " Nov 02 21:00:23.077430 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.077407 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4090b1-df6d-4014-8524-f39154343d5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4090b1-df6d-4014-8524-f39154343d5b-maas-api-service-ca\") pod \"2f4090b1-df6d-4014-8524-f39154343d5b\" (UID: \"2f4090b1-df6d-4014-8524-f39154343d5b\") " Nov 02 21:00:23.078714 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.078691 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbdf\" (UniqueName: \"kubernetes.io/projected/ccbf204a-de19-4517-bc00-7bcc299496b3-kube-api-access-tsbdf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.078850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.073939 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a63e09-b535-4190-9a7f-f72258817ebc-maas-api-service-ca" pod "17a63e09-b535-4190-9a7f-f72258817ebc" (UID: "17a63e09-b535-4190-9a7f-f72258817ebc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:23.078976 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.075623 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33550673-a1d4-4cb5-9467-a20cc86f3823-maas-api-service-ca" pod "33550673-a1d4-4cb5-9467-a20cc86f3823" (UID: "33550673-a1d4-4cb5-9467-a20cc86f3823"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:23.079070 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.076150 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db514a1-d913-4b14-8e2c-0770313c6d9d-maas-api-service-ca" pod "2db514a1-d913-4b14-8e2c-0770313c6d9d" (UID: "2db514a1-d913-4b14-8e2c-0770313c6d9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:23.079206 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.077937 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4090b1-df6d-4014-8524-f39154343d5b-maas-api-service-ca" pod "2f4090b1-df6d-4014-8524-f39154343d5b" (UID: "2f4090b1-df6d-4014-8524-f39154343d5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:23.089600 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.089548 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33550673-a1d4-4cb5-9467-a20cc86f3823-kube-api-access-5bwr8" pod "33550673-a1d4-4cb5-9467-a20cc86f3823" (UID: "33550673-a1d4-4cb5-9467-a20cc86f3823"). InnerVolumeSpecName "kube-api-access-5bwr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:23.090045 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.089890 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a63e09-b535-4190-9a7f-f72258817ebc-kube-api-access-5scpf" pod "17a63e09-b535-4190-9a7f-f72258817ebc" (UID: "17a63e09-b535-4190-9a7f-f72258817ebc"). InnerVolumeSpecName "kube-api-access-5scpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:23.090045 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.090016 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4090b1-df6d-4014-8524-f39154343d5b-kube-api-access-ql5gs" pod "2f4090b1-df6d-4014-8524-f39154343d5b" (UID: "2f4090b1-df6d-4014-8524-f39154343d5b"). InnerVolumeSpecName "kube-api-access-ql5gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:23.097786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.097721 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db514a1-d913-4b14-8e2c-0770313c6d9d-kube-api-access-6bqfg" pod "2db514a1-d913-4b14-8e2c-0770313c6d9d" (UID: "2db514a1-d913-4b14-8e2c-0770313c6d9d"). InnerVolumeSpecName "kube-api-access-6bqfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181311 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scpf\" (UniqueName: \"kubernetes.io/projected/17a63e09-b535-4190-9a7f-f72258817ebc-kube-api-access-5scpf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181345 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bqfg\" (UniqueName: \"kubernetes.io/projected/2db514a1-d913-4b14-8e2c-0770313c6d9d-kube-api-access-6bqfg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181359 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4090b1-df6d-4014-8524-f39154343d5b-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181373 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a63e09-b535-4190-9a7f-f72258817ebc-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181386 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwr8\" (UniqueName: \"kubernetes.io/projected/33550673-a1d4-4cb5-9467-a20cc86f3823-kube-api-access-5bwr8\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181400 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5gs\" (UniqueName: \"kubernetes.io/projected/2f4090b1-df6d-4014-8524-f39154343d5b-kube-api-access-ql5gs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181414 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33550673-a1d4-4cb5-9467-a20cc86f3823-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.181515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.181427 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db514a1-d913-4b14-8e2c-0770313c6d9d-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.220965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.218004 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" event={"ID":"2db514a1-d913-4b14-8e2c-0770313c6d9d","Type":"ContainerDied","Data":"8b22ba83bd1dead8bad328c8b8d01d28e393307c148f3578e92c7220715ea246"} Nov 02 21:00:23.220965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.218075 5800 scope.go:122] "RemoveContainer" containerID="d091835a239025220f0ddcda4598c95f3b17743244ce75478a5b1d362141ab64" Nov 02 21:00:23.220965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.218195 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh" Nov 02 21:00:23.241694 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.241659 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" event={"ID":"ccbf204a-de19-4517-bc00-7bcc299496b3","Type":"ContainerDied","Data":"184082b18dab245c4256456246dd7a2915e33eea91c882a73892faccf82515eb"} Nov 02 21:00:23.241982 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.241961 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc" Nov 02 21:00:23.266458 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.266423 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" event={"ID":"33550673-a1d4-4cb5-9467-a20cc86f3823","Type":"ContainerDied","Data":"b508aba616e5e1e0dd9601f923ca79744dc749a4924636194bbff7a4a7ecdbc3"} Nov 02 21:00:23.266767 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.266748 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj" Nov 02 21:00:23.273316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.271160 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" event={"ID":"2f4090b1-df6d-4014-8524-f39154343d5b","Type":"ContainerDied","Data":"df3650456e538197173e489d4f2a440f839ca2fd2c1697b63ec02eda6a26267a"} Nov 02 21:00:23.273316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.271236 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv" Nov 02 21:00:23.273316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.272682 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:23.281736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.281342 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc" Nov 02 21:00:23.282036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.282021 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" Nov 02 21:00:23.282583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.282561 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x" event={"ID":"17a63e09-b535-4190-9a7f-f72258817ebc","Type":"ContainerDied","Data":"56d052deb24295696c6761a3748409223ebd5a32557cda7f0b265da37c232179"} Nov 02 21:00:23.354504 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.354476 5800 scope.go:122] "RemoveContainer" containerID="be86376d969b35fbcab466710ce1c602c9d0fbc7d7e49f8064bbe5beb323d328" Nov 02 21:00:23.373494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.373460 5800 scope.go:122] "RemoveContainer" containerID="b9831aa1a1d0ea20445f10aa6041659cb62b388c64a8362a5ee45edefe7f4950" Nov 02 21:00:23.384129 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.384089 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-kube-api-access-qnh2m\" (UniqueName: \"kubernetes.io/projected/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-kube-api-access-qnh2m\") pod \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " Nov 02 21:00:23.384293 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.384201 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-maas-api-service-ca\") pod \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\" (UID: \"7fe0e8ef-ce92-40c0-89d4-7c1d91018892\") " Nov 02 21:00:23.385511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.385154 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-maas-api-service-ca" pod "7fe0e8ef-ce92-40c0-89d4-7c1d91018892" (UID: "7fe0e8ef-ce92-40c0-89d4-7c1d91018892"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:23.391022 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.390993 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-kube-api-access-qnh2m" pod "7fe0e8ef-ce92-40c0-89d4-7c1d91018892" (UID: "7fe0e8ef-ce92-40c0-89d4-7c1d91018892"). InnerVolumeSpecName "kube-api-access-qnh2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:23.392225 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.392118 5800 scope.go:122] "RemoveContainer" containerID="50e74ad89f6228b29538cfbf78893accb5079f796f2d3a79a4f91b4d16205cfb" Nov 02 21:00:23.410855 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.410732 5800 scope.go:122] "RemoveContainer" containerID="73c85462402b11573cd1a129cb83446991b3539328c0ab95e0ca7a52cc0ff847" Nov 02 21:00:23.486700 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.486666 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.486930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.486910 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnh2m\" (UniqueName: \"kubernetes.io/projected/7fe0e8ef-ce92-40c0-89d4-7c1d91018892-kube-api-access-qnh2m\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:23.590418 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.590348 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj"] Nov 02 21:00:23.596615 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.596574 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4530c5-29894220-zbgzj"] Nov 02 21:00:23.638349 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.638314 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" path="/var/lib/kubelet/pods/33550673-a1d4-4cb5-9467-a20cc86f3823/volumes" Nov 02 21:00:23.639242 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.639215 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" path="/var/lib/kubelet/pods/529e13a2-3e07-4af5-aaea-c3942a800589/volumes" Nov 02 21:00:23.639971 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.639950 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" path="/var/lib/kubelet/pods/5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939/volumes" Nov 02 21:00:23.869402 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.868847 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh"] Nov 02 21:00:23.876122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:23.876090 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894220-2b8qh"] Nov 02 21:00:24.411305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:24.410676 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc"] Nov 02 21:00:24.931540 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:24.931373 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" event={"ID":"7fe0e8ef-ce92-40c0-89d4-7c1d91018892","Type":"ContainerDied","Data":"9ee391b7b275b223e6e31ca3bb1941825bd83053eafe0ca29b652be90d81e4a8"} Nov 02 21:00:24.932034 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:24.931992 5800 scope.go:122] "RemoveContainer" containerID="67d946516e3c4c249c907109fb294347c562385450ae29279e020bae33d51c47" Nov 02 21:00:25.067237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:24.981799 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx" Nov 02 21:00:25.076809 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.076779 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-801923-29894220-2wcjc"] Nov 02 21:00:25.329357 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.329315 5800 scope.go:122] "RemoveContainer" containerID="fbb2f73a87b18ea78feafbbb378125785c05b9b8c18f56668294de9f2d0d5856" Nov 02 21:00:25.394307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.342608 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x"] Nov 02 21:00:25.394307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.377362 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-09ecd0-29894220-nv67x"] Nov 02 21:00:25.434440 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.434399 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv"] Nov 02 21:00:25.435065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.435043 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f3109b-29894220-nhpfv"] Nov 02 21:00:25.452307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.452245 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc"] Nov 02 21:00:25.452603 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.452582 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894220-5w5rc"] Nov 02 21:00:25.452748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.452734 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx"] Nov 02 21:00:25.458422 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.458389 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-14d090-29894220-t8zcx"] Nov 02 21:00:25.477184 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.477157 5800 scope.go:122] "RemoveContainer" containerID="e13dc62914db49202d7d3e7ccec3b71d996e1322cbdd66927ca6049291d98efa" Nov 02 21:00:25.532551 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.532430 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 02 21:00:25.657025 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.656991 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" path="/var/lib/kubelet/pods/17a63e09-b535-4190-9a7f-f72258817ebc/volumes" Nov 02 21:00:25.657920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.657893 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" path="/var/lib/kubelet/pods/2db514a1-d913-4b14-8e2c-0770313c6d9d/volumes" Nov 02 21:00:25.658697 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.658676 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" path="/var/lib/kubelet/pods/2f4090b1-df6d-4014-8524-f39154343d5b/volumes" Nov 02 21:00:25.687880 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.687768 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" path="/var/lib/kubelet/pods/7fe0e8ef-ce92-40c0-89d4-7c1d91018892/volumes" Nov 02 21:00:25.688872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.688850 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" path="/var/lib/kubelet/pods/ccbf204a-de19-4517-bc00-7bcc299496b3/volumes" Nov 02 21:00:25.689695 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.689675 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" path="/var/lib/kubelet/pods/de5226d7-33f1-41a5-a919-1c1bfc55962c/volumes" Nov 02 21:00:26.012812 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:25.979205 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:00:26.056995 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:26.039122 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:00:40.362690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.362650 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886"] Nov 02 21:00:40.363683 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363661 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.363752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363686 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.363752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363703 5800 cpu_manager.go:414] "containerMap: removing container" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:40.363752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363713 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:40.363752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363726 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.363752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363734 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.363752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363748 5800 cpu_manager.go:414] "containerMap: removing container" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363757 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363769 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363776 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363790 5800 cpu_manager.go:414] "containerMap: removing container" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363798 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363809 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363816 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363827 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363834 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363845 5800 cpu_manager.go:414] "containerMap: removing container" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363852 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363865 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363872 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363885 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363893 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363905 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363912 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363921 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363927 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363944 5800 cpu_manager.go:414] "containerMap: removing container" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:40.363955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363951 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363966 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363973 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363982 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.363989 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364007 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364014 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364023 5800 cpu_manager.go:414] "containerMap: removing container" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364031 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364045 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364052 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364062 5800 cpu_manager.go:414] "containerMap: removing container" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364068 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364083 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364089 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364101 5800 cpu_manager.go:414] "containerMap: removing container" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364108 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364117 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364124 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364134 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364141 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364152 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364159 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364175 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364182 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364191 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364198 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364213 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364221 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364233 5800 cpu_manager.go:414] "containerMap: removing container" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364242 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364259 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364266 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364296 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364304 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364318 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364325 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364337 5800 cpu_manager.go:414] "containerMap: removing container" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364344 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364383 5800 cpu_manager.go:414] "containerMap: removing container" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364392 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364405 5800 cpu_manager.go:414] "containerMap: removing container" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364414 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364426 5800 cpu_manager.go:414] "containerMap: removing container" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364433 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364442 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364450 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364612 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364622 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364634 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.364625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364646 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364655 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364664 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364672 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364681 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364689 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364697 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364705 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364714 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364723 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364744 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364752 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364761 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364768 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364775 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364784 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364791 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364798 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4baedbb3-e6fa-4a67-8e7f-a27005a97712" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364810 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37f3722-ebfe-4975-8b8f-6cd72a3c473e" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364817 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0e8ef-ce92-40c0-89d4-7c1d91018892" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364826 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364836 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364845 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364853 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364863 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364874 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="33550673-a1d4-4cb5-9467-a20cc86f3823" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364884 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:40.366174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.364893 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db514a1-d913-4b14-8e2c-0770313c6d9d" containerName="cleanup" Nov 02 21:00:40.369241 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.369220 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.417560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.417523 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886"] Nov 02 21:00:40.454977 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.454921 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4kz\" (UniqueName: \"kubernetes.io/projected/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-kube-api-access-4v4kz\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.455192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.455156 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.455368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.455348 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.459250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.459223 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:00:40.459417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.459239 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:00:40.459417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.459320 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:00:40.459417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.459397 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:00:40.557059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.557020 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.557256 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.557177 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.557368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.557311 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4kz\" (UniqueName: \"kubernetes.io/projected/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-kube-api-access-4v4kz\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.557945 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.557919 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.559636 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.559604 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.612941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.612865 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4kz\" (UniqueName: \"kubernetes.io/projected/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-kube-api-access-4v4kz\") pod \"maas-api-revoke-keys-e2e-inf-a-4530c5-zr886\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:40.683591 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:40.683557 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:41.076332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:41.065370 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886"] Nov 02 21:00:41.076332 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:00:41.065648 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9219a7_0b27_4399_8fc8_90b7bf3b635a.slice/crio-bbca7e6605c12124270aa343f47090af6ecdfb4a3f504a4b340f9924f3cb61ec WatchSource:0}: Error finding container bbca7e6605c12124270aa343f47090af6ecdfb4a3f504a4b340f9924f3cb61ec: Status 404 returned error can't find the container with id bbca7e6605c12124270aa343f47090af6ecdfb4a3f504a4b340f9924f3cb61ec Nov 02 21:00:41.350326 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:41.350211 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" event={"ID":"ec9219a7-0b27-4399-8fc8-90b7bf3b635a","Type":"ContainerStarted","Data":"bbca7e6605c12124270aa343f47090af6ecdfb4a3f504a4b340f9924f3cb61ec"} Nov 02 21:00:42.361501 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:42.361417 5800 generic.go:356] "Generic (PLEG): container finished" podID="ec9219a7-0b27-4399-8fc8-90b7bf3b635a" containerID="c24ea7bb02721c45baa3ace77b1b00b1b13d0053718b22e5b4078b90bf920a2c" exitCode=0 Nov 02 21:00:42.361890 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:42.361484 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" event={"ID":"ec9219a7-0b27-4399-8fc8-90b7bf3b635a","Type":"ContainerDied","Data":"c24ea7bb02721c45baa3ace77b1b00b1b13d0053718b22e5b4078b90bf920a2c"} Nov 02 21:00:43.560934 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.560908 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:43.697225 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.697144 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-openshift-ca\") pod \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " Nov 02 21:00:43.697393 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.697355 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-xks-ca\") pod \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " Nov 02 21:00:43.697591 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.697564 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-openshift-ca" pod "ec9219a7-0b27-4399-8fc8-90b7bf3b635a" (UID: "ec9219a7-0b27-4399-8fc8-90b7bf3b635a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:43.697591 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.697584 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-kube-api-access-4v4kz\" (UniqueName: \"kubernetes.io/projected/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-kube-api-access-4v4kz\") pod \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\" (UID: \"ec9219a7-0b27-4399-8fc8-90b7bf3b635a\") " Nov 02 21:00:43.698073 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.698049 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:43.699590 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.699567 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-xks-ca" pod "ec9219a7-0b27-4399-8fc8-90b7bf3b635a" (UID: "ec9219a7-0b27-4399-8fc8-90b7bf3b635a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:00:43.700312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.700265 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-kube-api-access-4v4kz" pod "ec9219a7-0b27-4399-8fc8-90b7bf3b635a" (UID: "ec9219a7-0b27-4399-8fc8-90b7bf3b635a"). InnerVolumeSpecName "kube-api-access-4v4kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:43.799760 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.799725 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v4kz\" (UniqueName: \"kubernetes.io/projected/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-kube-api-access-4v4kz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:43.799760 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:43.799756 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9219a7-0b27-4399-8fc8-90b7bf3b635a-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:44.379580 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:44.379548 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" Nov 02 21:00:44.379580 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:44.379569 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886" event={"ID":"ec9219a7-0b27-4399-8fc8-90b7bf3b635a","Type":"ContainerDied","Data":"bbca7e6605c12124270aa343f47090af6ecdfb4a3f504a4b340f9924f3cb61ec"} Nov 02 21:00:44.379794 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:44.379612 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbca7e6605c12124270aa343f47090af6ecdfb4a3f504a4b340f9924f3cb61ec" Nov 02 21:00:52.697783 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:00:52.697744 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:00:52.968101 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968024 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl"] Nov 02 21:00:52.968726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968711 5800 cpu_manager.go:414] "containerMap: removing container" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:52.968781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968727 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:52.968781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968750 5800 cpu_manager.go:414] "containerMap: removing container" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:52.968781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968756 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:52.968781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968767 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ec9219a7-0b27-4399-8fc8-90b7bf3b635a" containerName="revoke-keys" Nov 02 21:00:52.968781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968773 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9219a7-0b27-4399-8fc8-90b7bf3b635a" containerName="revoke-keys" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968877 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4090b1-df6d-4014-8524-f39154343d5b" containerName="cleanup" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968891 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3839a4-e546-4eed-b834-a62cd8b1f41c" containerName="cleanup" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968899 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5226d7-33f1-41a5-a919-1c1bfc55962c" containerName="cleanup" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968909 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="529e13a2-3e07-4af5-aaea-c3942a800589" containerName="cleanup" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968916 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0d3f5d-938b-4bc1-b4fa-6c8d168fe939" containerName="cleanup" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968923 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbf204a-de19-4517-bc00-7bcc299496b3" containerName="cleanup" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968931 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a63e09-b535-4190-9a7f-f72258817ebc" containerName="cleanup" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968937 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9219a7-0b27-4399-8fc8-90b7bf3b635a" containerName="revoke-keys" Nov 02 21:00:52.968952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.968942 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fbf823-6929-493a-911c-3404641c8863" containerName="cleanup" Nov 02 21:00:52.972848 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:52.972828 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.022419 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.022385 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:00:53.051769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.051730 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmrc\" (UniqueName: \"kubernetes.io/projected/edf332ea-b227-4da2-aa35-8bc8011909a8-kube-api-access-5qmrc\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.051987 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.051854 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.052127 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.052095 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.068696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.068654 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:00:53.068696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.068688 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:00:53.068932 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.068658 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:00:53.072553 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.072527 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl"] Nov 02 21:00:53.153362 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.153329 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.153546 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.153379 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmrc\" (UniqueName: \"kubernetes.io/projected/edf332ea-b227-4da2-aa35-8bc8011909a8-kube-api-access-5qmrc\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.153546 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.153491 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.154353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.154308 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.155997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.155973 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.206419 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.206386 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmrc\" (UniqueName: \"kubernetes.io/projected/edf332ea-b227-4da2-aa35-8bc8011909a8-kube-api-access-5qmrc\") pod \"maas-api-revoke-sub-a6e5ca2b-mxkgl\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.287526 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.287457 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:53.671114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:53.671078 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl"] Nov 02 21:00:53.672259 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:00:53.672220 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf332ea_b227_4da2_aa35_8bc8011909a8.slice/crio-cfee563952533ca8fba460f518dd64e48c629d38777c0a1eb00351208e5570c0 WatchSource:0}: Error finding container cfee563952533ca8fba460f518dd64e48c629d38777c0a1eb00351208e5570c0: Status 404 returned error can't find the container with id cfee563952533ca8fba460f518dd64e48c629d38777c0a1eb00351208e5570c0 Nov 02 21:00:54.268310 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.268263 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9"] Nov 02 21:00:54.272681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.272663 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.324447 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.324411 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9"] Nov 02 21:00:54.369322 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.369261 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rzdf\" (UniqueName: \"kubernetes.io/projected/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-kube-api-access-5rzdf\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.369508 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.369411 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.369508 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.369464 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.454878 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.454837 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" event={"ID":"edf332ea-b227-4da2-aa35-8bc8011909a8","Type":"ContainerStarted","Data":"88eccd9606044fb5361f21aa7535f64f38b3858690d64bf2414c0fb7cdcd8718"} Nov 02 21:00:54.455068 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.454898 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" event={"ID":"edf332ea-b227-4da2-aa35-8bc8011909a8","Type":"ContainerStarted","Data":"cfee563952533ca8fba460f518dd64e48c629d38777c0a1eb00351208e5570c0"} Nov 02 21:00:54.470989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.470946 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rzdf\" (UniqueName: \"kubernetes.io/projected/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-kube-api-access-5rzdf\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.471192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.471052 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.471192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.471110 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.471827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.471807 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.473525 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.473501 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.524045 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.523958 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rzdf\" (UniqueName: \"kubernetes.io/projected/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-kube-api-access-5rzdf\") pod \"maas-api-revoke-sub-43a981aa-gsxl9\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.561099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.561018 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" podStartSLOduration=2.561000622 podStartE2EDuration="2.561000622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:54.560445071 +0000 UTC m=+4353.077217659" watchObservedRunningTime="2026-11-02 21:00:54.561000622 +0000 UTC m=+4353.077773209" Nov 02 21:00:54.588133 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.588097 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:54.965572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:54.965542 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9"] Nov 02 21:00:54.966200 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:00:54.966171 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ccbe6f_4e1e_4b15_8b52_8a6c4b7b8df7.slice/crio-ec95752b0a76dd6857dca2bbdaff580a097ff61884f2c47c28240cb55826940e WatchSource:0}: Error finding container ec95752b0a76dd6857dca2bbdaff580a097ff61884f2c47c28240cb55826940e: Status 404 returned error can't find the container with id ec95752b0a76dd6857dca2bbdaff580a097ff61884f2c47c28240cb55826940e Nov 02 21:00:55.464994 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:55.464927 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" event={"ID":"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7","Type":"ContainerStarted","Data":"1e6ba3ae2498fdacb057b6e759d5652fb18ab760f1e58bc32fcf14fd492853a3"} Nov 02 21:00:55.465511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:55.465037 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" event={"ID":"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7","Type":"ContainerStarted","Data":"ec95752b0a76dd6857dca2bbdaff580a097ff61884f2c47c28240cb55826940e"} Nov 02 21:00:55.466897 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:55.466866 5800 generic.go:356] "Generic (PLEG): container finished" podID="edf332ea-b227-4da2-aa35-8bc8011909a8" containerID="88eccd9606044fb5361f21aa7535f64f38b3858690d64bf2414c0fb7cdcd8718" exitCode=0 Nov 02 21:00:55.467031 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:55.466914 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" event={"ID":"edf332ea-b227-4da2-aa35-8bc8011909a8","Type":"ContainerDied","Data":"88eccd9606044fb5361f21aa7535f64f38b3858690d64bf2414c0fb7cdcd8718"} Nov 02 21:00:55.572370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:55.572298 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" podStartSLOduration=1.572267461 podStartE2EDuration="1.572267461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:55.571421684 +0000 UTC m=+4354.088194269" watchObservedRunningTime="2026-11-02 21:00:55.572267461 +0000 UTC m=+4354.089040047" Nov 02 21:00:56.477122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.477082 5800 generic.go:356] "Generic (PLEG): container finished" podID="39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" containerID="1e6ba3ae2498fdacb057b6e759d5652fb18ab760f1e58bc32fcf14fd492853a3" exitCode=0 Nov 02 21:00:56.477619 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.477168 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" event={"ID":"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7","Type":"ContainerDied","Data":"1e6ba3ae2498fdacb057b6e759d5652fb18ab760f1e58bc32fcf14fd492853a3"} Nov 02 21:00:56.700306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.695149 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:56.807852 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.807820 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-openshift-ca\") pod \"edf332ea-b227-4da2-aa35-8bc8011909a8\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " Nov 02 21:00:56.808062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.808036 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf332ea-b227-4da2-aa35-8bc8011909a8-kube-api-access-5qmrc\" (UniqueName: \"kubernetes.io/projected/edf332ea-b227-4da2-aa35-8bc8011909a8-kube-api-access-5qmrc\") pod \"edf332ea-b227-4da2-aa35-8bc8011909a8\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " Nov 02 21:00:56.808136 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.808080 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-xks-ca\") pod \"edf332ea-b227-4da2-aa35-8bc8011909a8\" (UID: \"edf332ea-b227-4da2-aa35-8bc8011909a8\") " Nov 02 21:00:56.808207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.808181 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-openshift-ca" pod "edf332ea-b227-4da2-aa35-8bc8011909a8" (UID: "edf332ea-b227-4da2-aa35-8bc8011909a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:56.808801 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.808770 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:56.810353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.810321 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-xks-ca" pod "edf332ea-b227-4da2-aa35-8bc8011909a8" (UID: "edf332ea-b227-4da2-aa35-8bc8011909a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:00:56.810882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.810853 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf332ea-b227-4da2-aa35-8bc8011909a8-kube-api-access-5qmrc" pod "edf332ea-b227-4da2-aa35-8bc8011909a8" (UID: "edf332ea-b227-4da2-aa35-8bc8011909a8"). InnerVolumeSpecName "kube-api-access-5qmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:56.910181 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.910140 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmrc\" (UniqueName: \"kubernetes.io/projected/edf332ea-b227-4da2-aa35-8bc8011909a8-kube-api-access-5qmrc\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:56.910181 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:56.910179 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf332ea-b227-4da2-aa35-8bc8011909a8-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:57.487520 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.487487 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" Nov 02 21:00:57.487977 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.487526 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl" event={"ID":"edf332ea-b227-4da2-aa35-8bc8011909a8","Type":"ContainerDied","Data":"cfee563952533ca8fba460f518dd64e48c629d38777c0a1eb00351208e5570c0"} Nov 02 21:00:57.487977 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.487588 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfee563952533ca8fba460f518dd64e48c629d38777c0a1eb00351208e5570c0" Nov 02 21:00:57.661721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.661694 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:57.720794 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.720762 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-openshift-ca\") pod \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " Nov 02 21:00:57.721009 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.720807 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-xks-ca\") pod \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " Nov 02 21:00:57.721095 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.721013 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-kube-api-access-5rzdf\" (UniqueName: \"kubernetes.io/projected/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-kube-api-access-5rzdf\") pod \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\" (UID: \"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7\") " Nov 02 21:00:57.721210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.721187 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-openshift-ca" pod "39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" (UID: "39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:57.721664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.721642 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:57.723091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.723065 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-kube-api-access-5rzdf" pod "39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" (UID: "39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7"). InnerVolumeSpecName "kube-api-access-5rzdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:57.723216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.723176 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-xks-ca" pod "39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" (UID: "39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:00:57.822812 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.822767 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rzdf\" (UniqueName: \"kubernetes.io/projected/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-kube-api-access-5rzdf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:57.822812 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:57.822799 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:58.496668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:58.496630 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" Nov 02 21:00:58.497074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:58.496634 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9" event={"ID":"39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7","Type":"ContainerDied","Data":"ec95752b0a76dd6857dca2bbdaff580a097ff61884f2c47c28240cb55826940e"} Nov 02 21:00:58.497074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:00:58.496756 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec95752b0a76dd6857dca2bbdaff580a097ff61884f2c47c28240cb55826940e" Nov 02 21:01:12.229467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.229426 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg"] Nov 02 21:01:12.230138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.230120 5800 cpu_manager.go:414] "containerMap: removing container" podUID="39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" containerName="revoke-keys" Nov 02 21:01:12.230138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.230138 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" containerName="revoke-keys" Nov 02 21:01:12.230241 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.230188 5800 cpu_manager.go:414] "containerMap: removing container" podUID="edf332ea-b227-4da2-aa35-8bc8011909a8" containerName="revoke-keys" Nov 02 21:01:12.230241 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.230197 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf332ea-b227-4da2-aa35-8bc8011909a8" containerName="revoke-keys" Nov 02 21:01:12.230392 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.230375 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf332ea-b227-4da2-aa35-8bc8011909a8" containerName="revoke-keys" Nov 02 21:01:12.230437 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.230398 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" containerName="revoke-keys" Nov 02 21:01:12.234666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.234646 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.269937 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.269899 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-routing-overlay-volume\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.270117 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.269961 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btt9x\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-kube-api-access-btt9x\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.270117 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.270080 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-plugins-config-volume\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.270240 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.270227 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.287747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.287712 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg"] Nov 02 21:01:12.329820 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.329777 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 21:01:12.330008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.329905 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 21:01:12.330008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.329992 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 21:01:12.330166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.330090 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 21:01:12.330166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.330153 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-53672513306b\"" Nov 02 21:01:12.330250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.330243 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-bfm4n\"" Nov 02 21:01:12.371217 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.371174 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-plugins-config-volume\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.371453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.371406 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.371536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.371467 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-routing-overlay-volume\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.371536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.371510 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btt9x\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-kube-api-access-btt9x\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.372418 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.372382 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-plugins-config-volume\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.372790 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.372759 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-routing-overlay-volume\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.374715 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.374692 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.424946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.424904 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btt9x\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-kube-api-access-btt9x\") pod \"payload-processing-external-model-7bf47df886-rzkqg\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:12.552444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:12.552408 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:13.136800 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:01:13.136768 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95c3d3e_408c_4023_8968_26b565bba8f3.slice/crio-0ab65d7ee054549cb7a4bc9487e735f45e4878c6df8853baf11df0074f008720 WatchSource:0}: Error finding container 0ab65d7ee054549cb7a4bc9487e735f45e4878c6df8853baf11df0074f008720: Status 404 returned error can't find the container with id 0ab65d7ee054549cb7a4bc9487e735f45e4878c6df8853baf11df0074f008720 Nov 02 21:01:13.137492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:13.137460 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg"] Nov 02 21:01:13.753181 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:13.753130 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" event={"ID":"d95c3d3e-408c-4023-8968-26b565bba8f3","Type":"ContainerStarted","Data":"9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d"} Nov 02 21:01:13.753736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:13.753227 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" event={"ID":"d95c3d3e-408c-4023-8968-26b565bba8f3","Type":"ContainerStarted","Data":"0ab65d7ee054549cb7a4bc9487e735f45e4878c6df8853baf11df0074f008720"} Nov 02 21:01:13.753736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:13.753325 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:13.861649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:13.861592 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" podStartSLOduration=1.861578149 podStartE2EDuration="1.861578149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:01:13.859863755 +0000 UTC m=+4372.376636340" watchObservedRunningTime="2026-11-02 21:01:13.861578149 +0000 UTC m=+4372.378350737" Nov 02 21:01:14.918700 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:14.918656 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-53672513306b: secret "agc-cred-53672513306b" not found Nov 02 21:01:14.918700 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:14.918689 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg: secret "agc-cred-53672513306b" not found Nov 02 21:01:14.919207 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:14.918760 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials podName:d95c3d3e-408c-4023-8968-26b565bba8f3 nodeName:}" failed. No retries permitted until 2026-11-02 21:01:15.418743583 +0000 UTC m=+4373.935516163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials") pod "payload-processing-external-model-7bf47df886-rzkqg" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3") : secret "agc-cred-53672513306b" not found Nov 02 21:01:15.398515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:15.398446 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg"] Nov 02 21:01:15.425168 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:15.425136 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-53672513306b: secret "agc-cred-53672513306b" not found Nov 02 21:01:15.425168 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:15.425163 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg: secret "agc-cred-53672513306b" not found Nov 02 21:01:15.425411 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:15.425224 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials podName:d95c3d3e-408c-4023-8968-26b565bba8f3 nodeName:}" failed. No retries permitted until 2026-11-02 21:01:16.425208501 +0000 UTC m=+4374.941981078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials") pod "payload-processing-external-model-7bf47df886-rzkqg" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3") : secret "agc-cred-53672513306b" not found Nov 02 21:01:16.467617 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:16.467586 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-53672513306b: secret "agc-cred-53672513306b" not found Nov 02 21:01:16.467617 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:16.467612 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg: secret "agc-cred-53672513306b" not found Nov 02 21:01:16.468108 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:16.467671 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials podName:d95c3d3e-408c-4023-8968-26b565bba8f3 nodeName:}" failed. No retries permitted until 2026-11-02 21:01:18.467656344 +0000 UTC m=+4376.984428921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials") pod "payload-processing-external-model-7bf47df886-rzkqg" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3") : secret "agc-cred-53672513306b" not found Nov 02 21:01:18.497673 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:18.497632 5800 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-53672513306b: secret "agc-cred-53672513306b" not found Nov 02 21:01:18.497673 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:18.497662 5800 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg: secret "agc-cred-53672513306b" not found Nov 02 21:01:18.498199 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:18.497729 5800 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials podName:d95c3d3e-408c-4023-8968-26b565bba8f3 nodeName:}" failed. No retries permitted until 2026-11-02 21:01:22.497713003 +0000 UTC m=+4381.014485567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials") pod "payload-processing-external-model-7bf47df886-rzkqg" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3") : secret "agc-cred-53672513306b" not found Nov 02 21:01:20.773313 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:20.773254 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" podUID="d95c3d3e-408c-4023-8968-26b565bba8f3" containerName="payload-processing" containerID="cri-o://9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d" gracePeriod=25 Nov 02 21:01:21.129974 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.129941 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:21.262855 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.262799 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials\") pod \"d95c3d3e-408c-4023-8968-26b565bba8f3\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " Nov 02 21:01:21.263254 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.263221 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-routing-overlay-volume\") pod \"d95c3d3e-408c-4023-8968-26b565bba8f3\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " Nov 02 21:01:21.263545 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.263517 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-plugins-config-volume\") pod \"d95c3d3e-408c-4023-8968-26b565bba8f3\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " Nov 02 21:01:21.263775 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.263758 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-kube-api-access-btt9x\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-kube-api-access-btt9x\") pod \"d95c3d3e-408c-4023-8968-26b565bba8f3\" (UID: \"d95c3d3e-408c-4023-8968-26b565bba8f3\") " Nov 02 21:01:21.265236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.265191 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-routing-overlay-volume" pod "d95c3d3e-408c-4023-8968-26b565bba8f3" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:21.273233 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.265718 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-plugins-config-volume" pod "d95c3d3e-408c-4023-8968-26b565bba8f3" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:21.273233 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.268257 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-kube-api-access-btt9x" pod "d95c3d3e-408c-4023-8968-26b565bba8f3" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3"). InnerVolumeSpecName "kube-api-access-btt9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:21.309451 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.309359 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials" pod "d95c3d3e-408c-4023-8968-26b565bba8f3" (UID: "d95c3d3e-408c-4023-8968-26b565bba8f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:21.365663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.365560 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:21.365663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.365611 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btt9x\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-kube-api-access-btt9x\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:21.365663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.365627 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d95c3d3e-408c-4023-8968-26b565bba8f3-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:21.365663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.365642 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d95c3d3e-408c-4023-8968-26b565bba8f3-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:21.822582 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.822538 5800 generic.go:356] "Generic (PLEG): container finished" podID="d95c3d3e-408c-4023-8968-26b565bba8f3" containerID="9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d" exitCode=0 Nov 02 21:01:21.822582 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.822584 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" Nov 02 21:01:21.823167 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.822622 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" event={"ID":"d95c3d3e-408c-4023-8968-26b565bba8f3","Type":"ContainerDied","Data":"9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d"} Nov 02 21:01:21.823167 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.822686 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg" event={"ID":"d95c3d3e-408c-4023-8968-26b565bba8f3","Type":"ContainerDied","Data":"0ab65d7ee054549cb7a4bc9487e735f45e4878c6df8853baf11df0074f008720"} Nov 02 21:01:21.823167 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.822718 5800 scope.go:122] "RemoveContainer" containerID="9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d" Nov 02 21:01:21.839675 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.839644 5800 scope.go:122] "RemoveContainer" containerID="9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d" Nov 02 21:01:21.840133 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:21.840105 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d\": container with ID starting with 9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d not found: ID does not exist" containerID="9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d" Nov 02 21:01:21.840234 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.840144 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d"} err="failed to get container status \"9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d\": rpc error: code = NotFound desc = could not find container \"9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d\": container with ID starting with 9e4826560dffcf774c1921f7262167eb5a4e5e62e61ce2793344037285735a4d not found: ID does not exist" Nov 02 21:01:21.977227 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:21.977126 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg"] Nov 02 21:01:22.025083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.025035 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-rzkqg"] Nov 02 21:01:22.698185 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:22.698147 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:01:22.907221 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.907181 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-h47dg"] Nov 02 21:01:22.907928 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.907911 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d95c3d3e-408c-4023-8968-26b565bba8f3" containerName="payload-processing" Nov 02 21:01:22.907928 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.907928 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95c3d3e-408c-4023-8968-26b565bba8f3" containerName="payload-processing" Nov 02 21:01:22.908049 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.908039 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95c3d3e-408c-4023-8968-26b565bba8f3" containerName="payload-processing" Nov 02 21:01:22.913141 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.913119 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:22.964646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.964572 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-h47dg"] Nov 02 21:01:22.986877 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.986842 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm65l\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-kube-api-access-nm65l\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:22.987056 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.987028 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-provider-credentials\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:22.987254 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.987232 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-routing-overlay-volume\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:22.987332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:22.987298 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-plugins-config-volume\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.006639 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.006580 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 21:01:23.007039 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.006996 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 21:01:23.007178 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.007162 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 21:01:23.007247 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.007176 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 21:01:23.008511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.008492 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-hz97n\"" Nov 02 21:01:23.008511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.008505 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-53672513306b\"" Nov 02 21:01:23.089048 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.089011 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-provider-credentials\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.089239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.089130 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-routing-overlay-volume\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.089239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.089159 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-plugins-config-volume\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.089239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.089198 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm65l\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-kube-api-access-nm65l\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.089930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.089908 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-routing-overlay-volume\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.090019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.089981 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-plugins-config-volume\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.091623 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.091605 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-provider-credentials\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.142519 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.142483 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm65l\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-kube-api-access-nm65l\") pod \"payload-processing-external-model-7bf47df886-h47dg\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.228360 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.228243 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.624572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.624539 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95c3d3e-408c-4023-8968-26b565bba8f3" path="/var/lib/kubelet/pods/d95c3d3e-408c-4023-8968-26b565bba8f3/volumes" Nov 02 21:01:23.663544 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:01:23.663507 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db73376_cbc4_4791_81c4_4d9018fb4220.slice/crio-1557b7d1a521b41aee75e08db87c3090f357d4c64d63b5a5657c2999c2a51926 WatchSource:0}: Error finding container 1557b7d1a521b41aee75e08db87c3090f357d4c64d63b5a5657c2999c2a51926: Status 404 returned error can't find the container with id 1557b7d1a521b41aee75e08db87c3090f357d4c64d63b5a5657c2999c2a51926 Nov 02 21:01:23.663682 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.663543 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-h47dg"] Nov 02 21:01:23.841918 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.841878 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" event={"ID":"5db73376-cbc4-4791-81c4-4d9018fb4220","Type":"ContainerStarted","Data":"2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618"} Nov 02 21:01:23.842124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.841942 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:23.842124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.841958 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" event={"ID":"5db73376-cbc4-4791-81c4-4d9018fb4220","Type":"ContainerStarted","Data":"1557b7d1a521b41aee75e08db87c3090f357d4c64d63b5a5657c2999c2a51926"} Nov 02 21:01:23.949718 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:23.949584 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" podStartSLOduration=1.949563504 podStartE2EDuration="1.949563504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:01:23.948496615 +0000 UTC m=+4382.465269197" watchObservedRunningTime="2026-11-02 21:01:23.949563504 +0000 UTC m=+4382.466336089" Nov 02 21:01:27.263062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:27.263026 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-h47dg"] Nov 02 21:01:32.263570 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.263521 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" podUID="5db73376-cbc4-4791-81c4-4d9018fb4220" containerName="payload-processing" containerID="cri-o://2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618" gracePeriod=25 Nov 02 21:01:32.552790 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.552758 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:32.669996 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.669962 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-provider-credentials\") pod \"5db73376-cbc4-4791-81c4-4d9018fb4220\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " Nov 02 21:01:32.670202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.670038 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-plugins-config-volume\") pod \"5db73376-cbc4-4791-81c4-4d9018fb4220\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " Nov 02 21:01:32.670202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.670154 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-kube-api-access-nm65l\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-kube-api-access-nm65l\") pod \"5db73376-cbc4-4791-81c4-4d9018fb4220\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " Nov 02 21:01:32.670406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.670217 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-routing-overlay-volume\") pod \"5db73376-cbc4-4791-81c4-4d9018fb4220\" (UID: \"5db73376-cbc4-4791-81c4-4d9018fb4220\") " Nov 02 21:01:32.670842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.670811 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-plugins-config-volume" pod "5db73376-cbc4-4791-81c4-4d9018fb4220" (UID: "5db73376-cbc4-4791-81c4-4d9018fb4220"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:32.670973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.670847 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-routing-overlay-volume" pod "5db73376-cbc4-4791-81c4-4d9018fb4220" (UID: "5db73376-cbc4-4791-81c4-4d9018fb4220"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:01:32.673866 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.673840 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-provider-credentials" pod "5db73376-cbc4-4791-81c4-4d9018fb4220" (UID: "5db73376-cbc4-4791-81c4-4d9018fb4220"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:32.673978 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.673924 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-kube-api-access-nm65l" pod "5db73376-cbc4-4791-81c4-4d9018fb4220" (UID: "5db73376-cbc4-4791-81c4-4d9018fb4220"). InnerVolumeSpecName "kube-api-access-nm65l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:01:32.771958 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.771863 5800 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-provider-credentials\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:32.772215 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.772187 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:32.772215 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.772214 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm65l\" (UniqueName: \"kubernetes.io/projected/5db73376-cbc4-4791-81c4-4d9018fb4220-kube-api-access-nm65l\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:32.772401 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.772224 5800 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5db73376-cbc4-4791-81c4-4d9018fb4220-routing-overlay-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:01:32.909250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.909216 5800 generic.go:356] "Generic (PLEG): container finished" podID="5db73376-cbc4-4791-81c4-4d9018fb4220" containerID="2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618" exitCode=0 Nov 02 21:01:32.909473 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.909265 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" Nov 02 21:01:32.909473 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.909328 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" event={"ID":"5db73376-cbc4-4791-81c4-4d9018fb4220","Type":"ContainerDied","Data":"2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618"} Nov 02 21:01:32.909473 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.909399 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bf47df886-h47dg" event={"ID":"5db73376-cbc4-4791-81c4-4d9018fb4220","Type":"ContainerDied","Data":"1557b7d1a521b41aee75e08db87c3090f357d4c64d63b5a5657c2999c2a51926"} Nov 02 21:01:32.909473 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.909428 5800 scope.go:122] "RemoveContainer" containerID="2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618" Nov 02 21:01:32.921170 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.921153 5800 scope.go:122] "RemoveContainer" containerID="2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618" Nov 02 21:01:32.921471 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:32.921451 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618\": container with ID starting with 2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618 not found: ID does not exist" containerID="2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618" Nov 02 21:01:32.921567 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:32.921476 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618"} err="failed to get container status \"2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618\": rpc error: code = NotFound desc = could not find container \"2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618\": container with ID starting with 2c76213069b891bf9371a3b3934256ab6d8d17f54a6622da66fd4d3751936618 not found: ID does not exist" Nov 02 21:01:33.063872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:33.063774 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-h47dg"] Nov 02 21:01:33.109412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:33.109373 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bf47df886-h47dg"] Nov 02 21:01:33.623413 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:33.623371 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db73376-cbc4-4791-81c4-4d9018fb4220" path="/var/lib/kubelet/pods/5db73376-cbc4-4791-81c4-4d9018fb4220/volumes" Nov 02 21:01:48.495308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:48.494425 5800 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Nov 02 21:01:52.766856 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:01:52.766806 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:01:58.081206 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:58.081159 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72"] Nov 02 21:01:58.181517 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:58.181471 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv"] Nov 02 21:01:59.197906 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:59.197869 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n"] Nov 02 21:01:59.198512 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:01:59.198144 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" containerID="cri-o://5914595b7c768bf2560e3a9cf4e0e35a3b5ce1675ad7f8656c0fddc88bd4c91b" gracePeriod=30 Nov 02 21:02:03.082431 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.082333 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" podUID="c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" containerName="payload-processing" containerID="cri-o://c1a2bcf1f303eb44177000cfce731cf2d70da5399e87b879b11cb708c78ff03c" gracePeriod=25 Nov 02 21:02:03.181802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.181745 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" podUID="f260f4d0-4948-491a-b22e-772c461425b4" containerName="payload-pre-processing" containerID="cri-o://e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250" gracePeriod=25 Nov 02 21:02:03.212647 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:02:03.212618 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf260f4d0_4948_491a_b22e_772c461425b4.slice/crio-conmon-e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:02:03.238968 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.238921 5800 generic.go:356] "Generic (PLEG): container finished" podID="c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" containerID="c1a2bcf1f303eb44177000cfce731cf2d70da5399e87b879b11cb708c78ff03c" exitCode=0 Nov 02 21:02:03.242069 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.239322 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" event={"ID":"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4","Type":"ContainerDied","Data":"c1a2bcf1f303eb44177000cfce731cf2d70da5399e87b879b11cb708c78ff03c"} Nov 02 21:02:03.410409 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.410375 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 21:02:03.467070 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.467040 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-plugins-config-volume\") pod \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " Nov 02 21:02:03.467192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.467170 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-kube-api-access-bdjjw\") pod \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\" (UID: \"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4\") " Nov 02 21:02:03.467422 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.467397 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-plugins-config-volume" pod "c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" (UID: "c5a927c6-b344-4c65-8ce6-39a21a3f4fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:03.467823 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.467803 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:03.469207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.469186 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-kube-api-access-bdjjw" pod "c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" (UID: "c5a927c6-b344-4c65-8ce6-39a21a3f4fa4"). InnerVolumeSpecName "kube-api-access-bdjjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:03.489005 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.488981 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 21:02:03.568455 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.568415 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f260f4d0-4948-491a-b22e-772c461425b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f260f4d0-4948-491a-b22e-772c461425b4-plugins-config-volume\") pod \"f260f4d0-4948-491a-b22e-772c461425b4\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " Nov 02 21:02:03.568671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.568548 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f260f4d0-4948-491a-b22e-772c461425b4-kube-api-access-x4gct\" (UniqueName: \"kubernetes.io/projected/f260f4d0-4948-491a-b22e-772c461425b4-kube-api-access-x4gct\") pod \"f260f4d0-4948-491a-b22e-772c461425b4\" (UID: \"f260f4d0-4948-491a-b22e-772c461425b4\") " Nov 02 21:02:03.568893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.568863 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f260f4d0-4948-491a-b22e-772c461425b4-plugins-config-volume" pod "f260f4d0-4948-491a-b22e-772c461425b4" (UID: "f260f4d0-4948-491a-b22e-772c461425b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:03.569118 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.569096 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f260f4d0-4948-491a-b22e-772c461425b4-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:03.569208 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.569123 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdjjw\" (UniqueName: \"kubernetes.io/projected/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4-kube-api-access-bdjjw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:03.571250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.571220 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f260f4d0-4948-491a-b22e-772c461425b4-kube-api-access-x4gct" pod "f260f4d0-4948-491a-b22e-772c461425b4" (UID: "f260f4d0-4948-491a-b22e-772c461425b4"). InnerVolumeSpecName "kube-api-access-x4gct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:03.670889 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:03.670847 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4gct\" (UniqueName: \"kubernetes.io/projected/f260f4d0-4948-491a-b22e-772c461425b4-kube-api-access-x4gct\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.250462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.250428 5800 generic.go:356] "Generic (PLEG): container finished" podID="56831529-6563-43be-ac88-3166ffb219d2" containerID="5914595b7c768bf2560e3a9cf4e0e35a3b5ce1675ad7f8656c0fddc88bd4c91b" exitCode=0 Nov 02 21:02:04.250884 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.250505 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" event={"ID":"56831529-6563-43be-ac88-3166ffb219d2","Type":"ContainerDied","Data":"5914595b7c768bf2560e3a9cf4e0e35a3b5ce1675ad7f8656c0fddc88bd4c91b"} Nov 02 21:02:04.250884 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.250581 5800 scope.go:122] "RemoveContainer" containerID="da46f1a09931a4ce1a62c0c8377aaeb916b00d46eb084e739b4d7e17774564e5" Nov 02 21:02:04.253702 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.253680 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" Nov 02 21:02:04.253830 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.253706 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72" event={"ID":"c5a927c6-b344-4c65-8ce6-39a21a3f4fa4","Type":"ContainerDied","Data":"47d47e5e97c8c1b5fa7a02fe889d2cd6836223918addf48be6012a52a58ba377"} Nov 02 21:02:04.255475 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.255454 5800 generic.go:356] "Generic (PLEG): container finished" podID="f260f4d0-4948-491a-b22e-772c461425b4" containerID="e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250" exitCode=0 Nov 02 21:02:04.255611 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.255505 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" Nov 02 21:02:04.255611 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.255524 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" event={"ID":"f260f4d0-4948-491a-b22e-772c461425b4","Type":"ContainerDied","Data":"e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250"} Nov 02 21:02:04.255611 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.255565 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv" event={"ID":"f260f4d0-4948-491a-b22e-772c461425b4","Type":"ContainerDied","Data":"5ea8e83475565173aa40e9eb64c41a821e073fb67b76eb0a1c963f3e1ee5c184"} Nov 02 21:02:04.278390 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.278364 5800 scope.go:122] "RemoveContainer" containerID="c1a2bcf1f303eb44177000cfce731cf2d70da5399e87b879b11cb708c78ff03c" Nov 02 21:02:04.293687 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.293551 5800 scope.go:122] "RemoveContainer" containerID="e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250" Nov 02 21:02:04.306482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.306457 5800 scope.go:122] "RemoveContainer" containerID="e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250" Nov 02 21:02:04.306818 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:02:04.306794 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250\": container with ID starting with e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250 not found: ID does not exist" containerID="e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250" Nov 02 21:02:04.306919 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.306830 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250"} err="failed to get container status \"e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250\": rpc error: code = NotFound desc = could not find container \"e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250\": container with ID starting with e5f44bdfe226ec50979a02b00107946eb75db229ae9c828061d551a397fb2250 not found: ID does not exist" Nov 02 21:02:04.410765 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.410737 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72"] Nov 02 21:02:04.412333 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.412308 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 21:02:04.416314 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.416274 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e19d40-6f9d694f45-wbp72"] Nov 02 21:02:04.496221 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496191 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-credential-socket\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496261 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56831529-6563-43be-ac88-3166ffb219d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56831529-6563-43be-ac88-3166ffb219d2-istio-podinfo\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496321 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-certs\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496499 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496413 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-istio-token\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-istio-token\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496512 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-envoy\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496597 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496550 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-credential-socket" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:04.496597 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496578 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-kube-api-access-vrtjj\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-kube-api-access-vrtjj\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496620 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-data\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496664 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56831529-6563-43be-ac88-3166ffb219d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56831529-6563-43be-ac88-3166ffb219d2-istiod-ca-cert\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.496706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496686 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-certs" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:04.496879 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496732 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-socket\") pod \"56831529-6563-43be-ac88-3166ffb219d2\" (UID: \"56831529-6563-43be-ac88-3166ffb219d2\") " Nov 02 21:02:04.497016 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.496947 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-data" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:04.497098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.497043 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-socket" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:04.497157 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.497106 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56831529-6563-43be-ac88-3166ffb219d2-istiod-ca-cert" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:04.497336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.497310 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.497476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.497340 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.497476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.497353 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56831529-6563-43be-ac88-3166ffb219d2-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.497476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.497362 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.497476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.497371 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.498954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.498928 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56831529-6563-43be-ac88-3166ffb219d2-istio-podinfo" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:02:04.499058 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.498992 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-kube-api-access-vrtjj" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "kube-api-access-vrtjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:04.499143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.499126 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-envoy" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:02:04.499186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.499159 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-istio-token" pod "56831529-6563-43be-ac88-3166ffb219d2" (UID: "56831529-6563-43be-ac88-3166ffb219d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:04.515452 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.515430 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv"] Nov 02 21:02:04.520891 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.520868 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e19d40-7db94b69dc-pl9vv"] Nov 02 21:02:04.598934 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.598896 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrtjj\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-kube-api-access-vrtjj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.598934 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.598931 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56831529-6563-43be-ac88-3166ffb219d2-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.599154 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.598944 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56831529-6563-43be-ac88-3166ffb219d2-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:04.599154 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:04.598953 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56831529-6563-43be-ac88-3166ffb219d2-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:05.267367 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.267333 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" Nov 02 21:02:05.267886 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.267327 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n" event={"ID":"56831529-6563-43be-ac88-3166ffb219d2","Type":"ContainerDied","Data":"63dee6818a746626eaa8c5a3e9e6438170a302c74764d6558f437da85d3a354c"} Nov 02 21:02:05.267886 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.267482 5800 scope.go:122] "RemoveContainer" containerID="5914595b7c768bf2560e3a9cf4e0e35a3b5ce1675ad7f8656c0fddc88bd4c91b" Nov 02 21:02:05.423457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.423419 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n"] Nov 02 21:02:05.470482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.470443 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f3109b-openshift-default-979c959b5-bkh4n"] Nov 02 21:02:05.625149 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.625073 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56831529-6563-43be-ac88-3166ffb219d2" path="/var/lib/kubelet/pods/56831529-6563-43be-ac88-3166ffb219d2/volumes" Nov 02 21:02:05.625561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.625545 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" path="/var/lib/kubelet/pods/c5a927c6-b344-4c65-8ce6-39a21a3f4fa4/volumes" Nov 02 21:02:05.625907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:05.625894 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f260f4d0-4948-491a-b22e-772c461425b4" path="/var/lib/kubelet/pods/f260f4d0-4948-491a-b22e-772c461425b4/volumes" Nov 02 21:02:06.324978 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:06.324943 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg"] Nov 02 21:02:06.424577 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:06.424540 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h"] Nov 02 21:02:10.468348 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:10.465909 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886"] Nov 02 21:02:10.509251 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:10.509212 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4530c5-zr886"] Nov 02 21:02:11.325917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.325861 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" podUID="2347be97-13a0-4971-9f67-886f644c77a4" containerName="payload-processing" containerID="cri-o://891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39" gracePeriod=25 Nov 02 21:02:11.424856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.424807 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" podUID="7a2b299a-2dde-4d3b-bd34-0a527e4af184" containerName="payload-pre-processing" containerID="cri-o://a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f" gracePeriod=25 Nov 02 21:02:11.628994 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.628964 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9219a7-0b27-4399-8fc8-90b7bf3b635a" path="/var/lib/kubelet/pods/ec9219a7-0b27-4399-8fc8-90b7bf3b635a/volumes" Nov 02 21:02:11.635211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.635182 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 21:02:11.727249 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.727220 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2347be97-13a0-4971-9f67-886f644c77a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2347be97-13a0-4971-9f67-886f644c77a4-plugins-config-volume\") pod \"2347be97-13a0-4971-9f67-886f644c77a4\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " Nov 02 21:02:11.727501 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.727479 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2347be97-13a0-4971-9f67-886f644c77a4-kube-api-access-7gtn4\" (UniqueName: \"kubernetes.io/projected/2347be97-13a0-4971-9f67-886f644c77a4-kube-api-access-7gtn4\") pod \"2347be97-13a0-4971-9f67-886f644c77a4\" (UID: \"2347be97-13a0-4971-9f67-886f644c77a4\") " Nov 02 21:02:11.727711 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.727687 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2347be97-13a0-4971-9f67-886f644c77a4-plugins-config-volume" pod "2347be97-13a0-4971-9f67-886f644c77a4" (UID: "2347be97-13a0-4971-9f67-886f644c77a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:11.728044 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.728015 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2347be97-13a0-4971-9f67-886f644c77a4-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:11.729861 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.729836 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2347be97-13a0-4971-9f67-886f644c77a4-kube-api-access-7gtn4" pod "2347be97-13a0-4971-9f67-886f644c77a4" (UID: "2347be97-13a0-4971-9f67-886f644c77a4"). InnerVolumeSpecName "kube-api-access-7gtn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:11.750199 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.750162 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 21:02:11.829023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.828988 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2b299a-2dde-4d3b-bd34-0a527e4af184-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b299a-2dde-4d3b-bd34-0a527e4af184-plugins-config-volume\") pod \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " Nov 02 21:02:11.829223 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.829159 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2b299a-2dde-4d3b-bd34-0a527e4af184-kube-api-access-sgjxx\" (UniqueName: \"kubernetes.io/projected/7a2b299a-2dde-4d3b-bd34-0a527e4af184-kube-api-access-sgjxx\") pod \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\" (UID: \"7a2b299a-2dde-4d3b-bd34-0a527e4af184\") " Nov 02 21:02:11.829488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.829459 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2b299a-2dde-4d3b-bd34-0a527e4af184-plugins-config-volume" pod "7a2b299a-2dde-4d3b-bd34-0a527e4af184" (UID: "7a2b299a-2dde-4d3b-bd34-0a527e4af184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:11.829807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.829779 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gtn4\" (UniqueName: \"kubernetes.io/projected/2347be97-13a0-4971-9f67-886f644c77a4-kube-api-access-7gtn4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:11.829941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.829810 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b299a-2dde-4d3b-bd34-0a527e4af184-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:11.831526 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.831495 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2b299a-2dde-4d3b-bd34-0a527e4af184-kube-api-access-sgjxx" pod "7a2b299a-2dde-4d3b-bd34-0a527e4af184" (UID: "7a2b299a-2dde-4d3b-bd34-0a527e4af184"). InnerVolumeSpecName "kube-api-access-sgjxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:11.930820 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:11.930727 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgjxx\" (UniqueName: \"kubernetes.io/projected/7a2b299a-2dde-4d3b-bd34-0a527e4af184-kube-api-access-sgjxx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:12.323985 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.323951 5800 generic.go:356] "Generic (PLEG): container finished" podID="2347be97-13a0-4971-9f67-886f644c77a4" containerID="891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39" exitCode=0 Nov 02 21:02:12.324224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.323997 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" Nov 02 21:02:12.324224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.324043 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" event={"ID":"2347be97-13a0-4971-9f67-886f644c77a4","Type":"ContainerDied","Data":"891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39"} Nov 02 21:02:12.324224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.324103 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg" event={"ID":"2347be97-13a0-4971-9f67-886f644c77a4","Type":"ContainerDied","Data":"2613e5c30425731857a6f43a43f0261782bc3e8d1df6be85932fa629c5d64da1"} Nov 02 21:02:12.324224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.324134 5800 scope.go:122] "RemoveContainer" containerID="891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39" Nov 02 21:02:12.325705 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.325680 5800 generic.go:356] "Generic (PLEG): container finished" podID="7a2b299a-2dde-4d3b-bd34-0a527e4af184" containerID="a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f" exitCode=0 Nov 02 21:02:12.325834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.325730 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" Nov 02 21:02:12.325834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.325746 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" event={"ID":"7a2b299a-2dde-4d3b-bd34-0a527e4af184","Type":"ContainerDied","Data":"a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f"} Nov 02 21:02:12.325834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.325805 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h" event={"ID":"7a2b299a-2dde-4d3b-bd34-0a527e4af184","Type":"ContainerDied","Data":"680cab9dd24dbc3e5116b863a20d0a1910399e0b7a9d88cf0d14933b75917227"} Nov 02 21:02:12.341074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.340876 5800 scope.go:122] "RemoveContainer" containerID="891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39" Nov 02 21:02:12.341293 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:02:12.341256 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39\": container with ID starting with 891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39 not found: ID does not exist" containerID="891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39" Nov 02 21:02:12.341368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.341301 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39"} err="failed to get container status \"891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39\": rpc error: code = NotFound desc = could not find container \"891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39\": container with ID starting with 891bfe8d69d8952afae5945fd6d5152bf1d759d3d0130f96815fc20c4d038e39 not found: ID does not exist" Nov 02 21:02:12.341368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.341324 5800 scope.go:122] "RemoveContainer" containerID="a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f" Nov 02 21:02:12.355690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.355664 5800 scope.go:122] "RemoveContainer" containerID="a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f" Nov 02 21:02:12.356055 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:02:12.356031 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f\": container with ID starting with a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f not found: ID does not exist" containerID="a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f" Nov 02 21:02:12.356126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.356070 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f"} err="failed to get container status \"a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f\": rpc error: code = NotFound desc = could not find container \"a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f\": container with ID starting with a7a85899af059b5eb1a01e3b8fdbbe4edc8fff73caf5fc564a5a8c471900cc9f not found: ID does not exist" Nov 02 21:02:12.487384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.487344 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg"] Nov 02 21:02:12.535360 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.535325 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-801923-7bfbc8bff7-lzqsg"] Nov 02 21:02:12.635364 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.635253 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h"] Nov 02 21:02:12.640992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:12.640961 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-801923-697d9748cf-td79h"] Nov 02 21:02:13.623554 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:13.623512 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2347be97-13a0-4971-9f67-886f644c77a4" path="/var/lib/kubelet/pods/2347be97-13a0-4971-9f67-886f644c77a4/volumes" Nov 02 21:02:13.624054 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:13.624033 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2b299a-2dde-4d3b-bd34-0a527e4af184" path="/var/lib/kubelet/pods/7a2b299a-2dde-4d3b-bd34-0a527e4af184/volumes" Nov 02 21:02:16.348265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348228 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt"] Nov 02 21:02:16.348865 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348850 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" containerName="payload-processing" Nov 02 21:02:16.348909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348867 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" containerName="payload-processing" Nov 02 21:02:16.348909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348878 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7a2b299a-2dde-4d3b-bd34-0a527e4af184" containerName="payload-pre-processing" Nov 02 21:02:16.348909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348883 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2b299a-2dde-4d3b-bd34-0a527e4af184" containerName="payload-pre-processing" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348913 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f260f4d0-4948-491a-b22e-772c461425b4" containerName="payload-pre-processing" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348919 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f260f4d0-4948-491a-b22e-772c461425b4" containerName="payload-pre-processing" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348926 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5db73376-cbc4-4791-81c4-4d9018fb4220" containerName="payload-processing" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348932 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db73376-cbc4-4791-81c4-4d9018fb4220" containerName="payload-processing" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348942 5800 cpu_manager.go:414] "containerMap: removing container" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348947 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348971 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2347be97-13a0-4971-9f67-886f644c77a4" containerName="payload-processing" Nov 02 21:02:16.349020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.348975 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2347be97-13a0-4971-9f67-886f644c77a4" containerName="payload-processing" Nov 02 21:02:16.349305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.349061 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2347be97-13a0-4971-9f67-886f644c77a4" containerName="payload-processing" Nov 02 21:02:16.349305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.349071 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a927c6-b344-4c65-8ce6-39a21a3f4fa4" containerName="payload-processing" Nov 02 21:02:16.349305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.349078 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2b299a-2dde-4d3b-bd34-0a527e4af184" containerName="payload-pre-processing" Nov 02 21:02:16.349305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.349089 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f260f4d0-4948-491a-b22e-772c461425b4" containerName="payload-pre-processing" Nov 02 21:02:16.349305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.349095 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db73376-cbc4-4791-81c4-4d9018fb4220" containerName="payload-processing" Nov 02 21:02:16.349305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.349101 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" Nov 02 21:02:16.349305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.349109 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" Nov 02 21:02:16.354472 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.354448 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.391821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.391786 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.391991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.391857 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.391991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.391967 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsp5d\" (UniqueName: \"kubernetes.io/projected/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-kube-api-access-fsp5d\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.406488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.406456 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt"] Nov 02 21:02:16.449062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.449023 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:02:16.449274 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.449136 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:02:16.451520 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.451492 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:02:16.451728 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.451496 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:02:16.493338 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.493270 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.493534 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.493384 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsp5d\" (UniqueName: \"kubernetes.io/projected/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-kube-api-access-fsp5d\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.493534 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.493496 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.494126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.494090 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.495720 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.495700 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.546548 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.546517 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsp5d\" (UniqueName: \"kubernetes.io/projected/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-kube-api-access-fsp5d\") pod \"maas-api-revoke-sub-1b3a6345-6w8pt\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:16.668529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:16.668427 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:17.040119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:17.040084 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt"] Nov 02 21:02:17.041230 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:02:17.041198 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff3b757_8b28_4ae0_a030_5b5e3feef2bb.slice/crio-88302b6bbc30909284d4c5757cbe4db1dfc128387cd4973eeb7fa9ba3dd5a36a WatchSource:0}: Error finding container 88302b6bbc30909284d4c5757cbe4db1dfc128387cd4973eeb7fa9ba3dd5a36a: Status 404 returned error can't find the container with id 88302b6bbc30909284d4c5757cbe4db1dfc128387cd4973eeb7fa9ba3dd5a36a Nov 02 21:02:17.364706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:17.364663 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" event={"ID":"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb","Type":"ContainerStarted","Data":"aace42eea3a0b20a289a775d5bcff056fa36ad18af9c8c5d66f401e50a69090e"} Nov 02 21:02:17.365119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:17.364719 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" event={"ID":"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb","Type":"ContainerStarted","Data":"88302b6bbc30909284d4c5757cbe4db1dfc128387cd4973eeb7fa9ba3dd5a36a"} Nov 02 21:02:17.488275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:17.488218 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" podStartSLOduration=1.488203975 podStartE2EDuration="1.488203975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:02:17.488136731 +0000 UTC m=+4436.004909316" watchObservedRunningTime="2026-11-02 21:02:17.488203975 +0000 UTC m=+4436.004976556" Nov 02 21:02:18.375310 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:18.375248 5800 generic.go:356] "Generic (PLEG): container finished" podID="0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" containerID="aace42eea3a0b20a289a775d5bcff056fa36ad18af9c8c5d66f401e50a69090e" exitCode=0 Nov 02 21:02:18.375759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:18.375336 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" event={"ID":"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb","Type":"ContainerDied","Data":"aace42eea3a0b20a289a775d5bcff056fa36ad18af9c8c5d66f401e50a69090e"} Nov 02 21:02:19.556397 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.556366 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:19.632952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.632854 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-openshift-ca\") pod \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " Nov 02 21:02:19.633134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.632954 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-xks-ca\") pod \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " Nov 02 21:02:19.633134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.633008 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-kube-api-access-fsp5d\" (UniqueName: \"kubernetes.io/projected/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-kube-api-access-fsp5d\") pod \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\" (UID: \"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb\") " Nov 02 21:02:19.633316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.633244 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-openshift-ca" pod "0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" (UID: "0ff3b757-8b28-4ae0-a030-5b5e3feef2bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:02:19.633560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.633533 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:19.635339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.635303 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-xks-ca" pod "0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" (UID: "0ff3b757-8b28-4ae0-a030-5b5e3feef2bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:02:19.635456 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.635398 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-kube-api-access-fsp5d" pod "0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" (UID: "0ff3b757-8b28-4ae0-a030-5b5e3feef2bb"). InnerVolumeSpecName "kube-api-access-fsp5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:02:19.735277 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.735232 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsp5d\" (UniqueName: \"kubernetes.io/projected/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-kube-api-access-fsp5d\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:19.735277 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:19.735271 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:02:20.391178 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:20.391144 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" Nov 02 21:02:20.391406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:20.391179 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt" event={"ID":"0ff3b757-8b28-4ae0-a030-5b5e3feef2bb","Type":"ContainerDied","Data":"88302b6bbc30909284d4c5757cbe4db1dfc128387cd4973eeb7fa9ba3dd5a36a"} Nov 02 21:02:20.391406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:20.391239 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88302b6bbc30909284d4c5757cbe4db1dfc128387cd4973eeb7fa9ba3dd5a36a" Nov 02 21:02:22.697490 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:02:22.697450 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:02:45.433431 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:45.433376 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6" podUID="16c42059-424f-41a1-9108-ba08f2c79af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:02:45.622305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:45.619180 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 02 21:02:45.684376 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:45.661515 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:02:45.685024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:45.673256 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 02 21:02:45.754308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:02:45.750888 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:02:52.696670 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:02:52.696627 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:03:20.237187 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:20.237144 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5"] Nov 02 21:03:20.237768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:20.237437 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" containerID="cri-o://38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81" gracePeriod=30 Nov 02 21:03:22.494605 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:22.494572 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:03:22.498229 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:22.498204 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:03:22.702436 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:03:22.702394 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:03:25.515242 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.515217 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 21:03:25.664605 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.664564 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e096d67b-03cb-45da-b019-d36fe8dc3f97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096d67b-03cb-45da-b019-d36fe8dc3f97-istiod-ca-cert\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.664802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.664635 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-envoy\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.664802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.664720 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-data\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.664802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.664791 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-socket\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.664992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.664827 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-token\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-token\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.664992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.664902 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-podinfo\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.664992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.664978 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-kube-api-access-7fls5\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-kube-api-access-7fls5\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.665100 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.665064 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-certs\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.665155 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.665128 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-credential-socket\") pod \"e096d67b-03cb-45da-b019-d36fe8dc3f97\" (UID: \"e096d67b-03cb-45da-b019-d36fe8dc3f97\") " Nov 02 21:03:25.665808 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.665708 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-socket" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:03:25.666224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.666019 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e096d67b-03cb-45da-b019-d36fe8dc3f97-istiod-ca-cert" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:03:25.666587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.666397 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-credential-socket" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:03:25.666890 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.666729 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-certs" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:03:25.666973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.666895 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-data" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:03:25.667420 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.667394 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.667420 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.667422 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.667607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.667438 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e096d67b-03cb-45da-b019-d36fe8dc3f97-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.667607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.667452 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.667607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.667468 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.669595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.669466 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-kube-api-access-7fls5" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "kube-api-access-7fls5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:03:25.669595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.669488 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-podinfo" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:03:25.669965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.669932 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-token" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:03:25.670498 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.670209 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-envoy" pod "e096d67b-03cb-45da-b019-d36fe8dc3f97" (UID: "e096d67b-03cb-45da-b019-d36fe8dc3f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:03:25.768210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.768124 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.768210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.768155 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.768210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.768164 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e096d67b-03cb-45da-b019-d36fe8dc3f97-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.768210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.768177 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fls5\" (UniqueName: \"kubernetes.io/projected/e096d67b-03cb-45da-b019-d36fe8dc3f97-kube-api-access-7fls5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:03:25.982905 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:25.982873 5800 scope.go:122] "RemoveContainer" containerID="baa17535f2aa6e65bf7de744efbe3b8c77f876d0c4349087b50ab2eed5076dcd" Nov 02 21:03:26.007109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.007077 5800 generic.go:356] "Generic (PLEG): container finished" podID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerID="38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81" exitCode=0 Nov 02 21:03:26.007268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.007129 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" Nov 02 21:03:26.007268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.007157 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" event={"ID":"e096d67b-03cb-45da-b019-d36fe8dc3f97","Type":"ContainerDied","Data":"38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81"} Nov 02 21:03:26.007268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.007208 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5" event={"ID":"e096d67b-03cb-45da-b019-d36fe8dc3f97","Type":"ContainerDied","Data":"c416d663c23541e3e9154d2388ebc4a06db72f1785f4ae9516c09296183e8cb3"} Nov 02 21:03:26.007268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.007231 5800 scope.go:122] "RemoveContainer" containerID="38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81" Nov 02 21:03:26.018311 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.018255 5800 scope.go:122] "RemoveContainer" containerID="38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81" Nov 02 21:03:26.018576 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:03:26.018557 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81\": container with ID starting with 38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81 not found: ID does not exist" containerID="38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81" Nov 02 21:03:26.018620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.018585 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81"} err="failed to get container status \"38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81\": rpc error: code = NotFound desc = could not find container \"38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81\": container with ID starting with 38fbba9e154a5dfb19c1b51a9a2ff245af08f3f3011f2cfa59c16220885b1a81 not found: ID does not exist" Nov 02 21:03:26.220350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.220313 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5"] Nov 02 21:03:26.225319 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:26.225272 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e19d40-openshift-default-78d549846c-6xhx5"] Nov 02 21:03:27.625191 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:27.625157 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" path="/var/lib/kubelet/pods/e096d67b-03cb-45da-b019-d36fe8dc3f97/volumes" Nov 02 21:03:52.696553 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:03:52.696510 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:03:57.900959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.900920 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb"] Nov 02 21:03:57.901604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901586 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" containerName="revoke-keys" Nov 02 21:03:57.901663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901606 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" containerName="revoke-keys" Nov 02 21:03:57.901663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901625 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" Nov 02 21:03:57.901663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901630 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" Nov 02 21:03:57.901663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901642 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" Nov 02 21:03:57.901663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901648 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" Nov 02 21:03:57.901835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901672 5800 cpu_manager.go:414] "containerMap: removing container" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" Nov 02 21:03:57.901835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901677 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56831529-6563-43be-ac88-3166ffb219d2" containerName="istio-proxy" Nov 02 21:03:57.901835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901795 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" containerName="revoke-keys" Nov 02 21:03:57.901835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901804 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" Nov 02 21:03:57.901835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.901810 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e096d67b-03cb-45da-b019-d36fe8dc3f97" containerName="istio-proxy" Nov 02 21:03:57.905494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.905476 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:57.956529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.956494 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb"] Nov 02 21:03:57.997079 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.997040 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:57.997271 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.997247 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:57.997390 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.997360 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l578\" (UniqueName: \"kubernetes.io/projected/f58d2e8b-d78b-4b04-859e-c1fe73611ada-kube-api-access-4l578\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:57.998627 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.998607 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:03:57.998732 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:57.998660 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:03:58.000542 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.000524 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:03:58.000638 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.000535 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:03:58.099095 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.099048 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:58.099364 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.099223 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:58.099364 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.099292 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l578\" (UniqueName: \"kubernetes.io/projected/f58d2e8b-d78b-4b04-859e-c1fe73611ada-kube-api-access-4l578\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:58.099924 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.099898 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:58.101647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.101628 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:58.152656 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.152571 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l578\" (UniqueName: \"kubernetes.io/projected/f58d2e8b-d78b-4b04-859e-c1fe73611ada-kube-api-access-4l578\") pod \"maas-api-revoke-sub-55bfd721-zgcrb\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:58.220814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.220772 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:03:58.595793 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:58.595762 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb"] Nov 02 21:03:58.596770 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:03:58.596736 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58d2e8b_d78b_4b04_859e_c1fe73611ada.slice/crio-c1603afd5919e10978a70e5703200c0e8bc6eea478e1e5ba1a0e25bfe5d314e7 WatchSource:0}: Error finding container c1603afd5919e10978a70e5703200c0e8bc6eea478e1e5ba1a0e25bfe5d314e7: Status 404 returned error can't find the container with id c1603afd5919e10978a70e5703200c0e8bc6eea478e1e5ba1a0e25bfe5d314e7 Nov 02 21:03:59.262591 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:59.262555 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" event={"ID":"f58d2e8b-d78b-4b04-859e-c1fe73611ada","Type":"ContainerStarted","Data":"6b5cdd89b1baffb8a1f446033e418bc3c05c8978416646f938cbd08045d3bdef"} Nov 02 21:03:59.262993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:59.262612 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" event={"ID":"f58d2e8b-d78b-4b04-859e-c1fe73611ada","Type":"ContainerStarted","Data":"c1603afd5919e10978a70e5703200c0e8bc6eea478e1e5ba1a0e25bfe5d314e7"} Nov 02 21:03:59.369639 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:03:59.369584 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" podStartSLOduration=2.369567879 podStartE2EDuration="2.369567879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:03:59.368147334 +0000 UTC m=+4537.884919921" watchObservedRunningTime="2026-11-02 21:03:59.369567879 +0000 UTC m=+4537.886340461" Nov 02 21:04:00.271595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:00.271556 5800 generic.go:356] "Generic (PLEG): container finished" podID="f58d2e8b-d78b-4b04-859e-c1fe73611ada" containerID="6b5cdd89b1baffb8a1f446033e418bc3c05c8978416646f938cbd08045d3bdef" exitCode=0 Nov 02 21:04:00.272055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:00.271646 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" event={"ID":"f58d2e8b-d78b-4b04-859e-c1fe73611ada","Type":"ContainerDied","Data":"6b5cdd89b1baffb8a1f446033e418bc3c05c8978416646f938cbd08045d3bdef"} Nov 02 21:04:01.488257 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.488230 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:04:01.548057 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.547970 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-openshift-ca\") pod \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " Nov 02 21:04:01.548234 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.548097 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-xks-ca\") pod \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " Nov 02 21:04:01.548234 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.548145 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58d2e8b-d78b-4b04-859e-c1fe73611ada-kube-api-access-4l578\" (UniqueName: \"kubernetes.io/projected/f58d2e8b-d78b-4b04-859e-c1fe73611ada-kube-api-access-4l578\") pod \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\" (UID: \"f58d2e8b-d78b-4b04-859e-c1fe73611ada\") " Nov 02 21:04:01.548434 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.548406 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-openshift-ca" pod "f58d2e8b-d78b-4b04-859e-c1fe73611ada" (UID: "f58d2e8b-d78b-4b04-859e-c1fe73611ada"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:04:01.548821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.548795 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:04:01.550326 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.550264 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-xks-ca" pod "f58d2e8b-d78b-4b04-859e-c1fe73611ada" (UID: "f58d2e8b-d78b-4b04-859e-c1fe73611ada"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:04:01.550465 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.550402 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58d2e8b-d78b-4b04-859e-c1fe73611ada-kube-api-access-4l578" pod "f58d2e8b-d78b-4b04-859e-c1fe73611ada" (UID: "f58d2e8b-d78b-4b04-859e-c1fe73611ada"). InnerVolumeSpecName "kube-api-access-4l578". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:04:01.650131 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.650095 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58d2e8b-d78b-4b04-859e-c1fe73611ada-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:04:01.650131 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:01.650126 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l578\" (UniqueName: \"kubernetes.io/projected/f58d2e8b-d78b-4b04-859e-c1fe73611ada-kube-api-access-4l578\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:04:02.287253 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:02.287214 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" event={"ID":"f58d2e8b-d78b-4b04-859e-c1fe73611ada","Type":"ContainerDied","Data":"c1603afd5919e10978a70e5703200c0e8bc6eea478e1e5ba1a0e25bfe5d314e7"} Nov 02 21:04:02.287253 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:02.287248 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb" Nov 02 21:04:02.287542 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:02.287264 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1603afd5919e10978a70e5703200c0e8bc6eea478e1e5ba1a0e25bfe5d314e7" Nov 02 21:04:22.696091 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:04:22.696052 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:04:52.895064 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:04:52.895018 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:04:53.155374 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:53.153397 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:04:53.155374 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:53.153483 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:04:53.584427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:53.584386 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:04:53.584725 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:04:53.584695 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:05:22.696644 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:05:22.696552 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:05:51.478402 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.478357 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt"] Nov 02 21:05:51.479445 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.479421 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f58d2e8b-d78b-4b04-859e-c1fe73611ada" containerName="revoke-keys" Nov 02 21:05:51.479445 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.479446 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58d2e8b-d78b-4b04-859e-c1fe73611ada" containerName="revoke-keys" Nov 02 21:05:51.479624 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.479610 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58d2e8b-d78b-4b04-859e-c1fe73611ada" containerName="revoke-keys" Nov 02 21:05:51.484613 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.484586 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:51.528932 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.528896 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt"] Nov 02 21:05:51.554645 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.554597 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86154ed-afba-4f0b-a598-a10a3dd9003e-plugins-config-volume\") pod \"payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:51.554830 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.554816 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fw8t\" (UniqueName: \"kubernetes.io/projected/d86154ed-afba-4f0b-a598-a10a3dd9003e-kube-api-access-6fw8t\") pod \"payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:51.569953 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.569905 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-68a6c923\"" Nov 02 21:05:51.569953 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.569912 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-68a6c923-dockercfg-w5h5g\"" Nov 02 21:05:51.657327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.656798 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fw8t\" (UniqueName: \"kubernetes.io/projected/d86154ed-afba-4f0b-a598-a10a3dd9003e-kube-api-access-6fw8t\") pod \"payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:51.657327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.657215 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86154ed-afba-4f0b-a598-a10a3dd9003e-plugins-config-volume\") pod \"payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:51.657964 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.657940 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86154ed-afba-4f0b-a598-a10a3dd9003e-plugins-config-volume\") pod \"payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:51.710048 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.710005 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fw8t\" (UniqueName: \"kubernetes.io/projected/d86154ed-afba-4f0b-a598-a10a3dd9003e-kube-api-access-6fw8t\") pod \"payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:51.799637 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:51.799599 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:52.266212 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:52.266180 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt"] Nov 02 21:05:52.268485 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:05:52.268452 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86154ed_afba_4f0b_a598_a10a3dd9003e.slice/crio-5054b2aa3f9ad1037e95a238749df30c6bb303741c96075204a8d83540fed1ac WatchSource:0}: Error finding container 5054b2aa3f9ad1037e95a238749df30c6bb303741c96075204a8d83540fed1ac: Status 404 returned error can't find the container with id 5054b2aa3f9ad1037e95a238749df30c6bb303741c96075204a8d83540fed1ac Nov 02 21:05:52.740430 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:05:52.740329 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:05:53.261010 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:53.260971 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" event={"ID":"d86154ed-afba-4f0b-a598-a10a3dd9003e","Type":"ContainerStarted","Data":"c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274"} Nov 02 21:05:53.261218 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:53.261030 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" event={"ID":"d86154ed-afba-4f0b-a598-a10a3dd9003e","Type":"ContainerStarted","Data":"5054b2aa3f9ad1037e95a238749df30c6bb303741c96075204a8d83540fed1ac"} Nov 02 21:05:53.261218 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:53.261072 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:05:53.370155 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:53.370093 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" podStartSLOduration=2.370077775 podStartE2EDuration="2.370077775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:53.365994444 +0000 UTC m=+4651.882767031" watchObservedRunningTime="2026-11-02 21:05:53.370077775 +0000 UTC m=+4651.886850359" Nov 02 21:05:57.043188 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:57.043133 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl"] Nov 02 21:05:57.089226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:57.089189 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6e5ca2b-mxkgl"] Nov 02 21:05:57.623907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:57.623873 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf332ea-b227-4da2-aa35-8bc8011909a8" path="/var/lib/kubelet/pods/edf332ea-b227-4da2-aa35-8bc8011909a8/volumes" Nov 02 21:05:58.075494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:58.075448 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9"] Nov 02 21:05:58.184323 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:58.183363 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43a981aa-gsxl9"] Nov 02 21:05:59.625466 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:05:59.625434 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7" path="/var/lib/kubelet/pods/39ccbe6f-4e1e-4b15-8b52-8a6c4b7b8df7/volumes" Nov 02 21:06:04.273384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:06:04.273349 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:06:22.695858 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:06:22.695817 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:06:26.034641 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:06:26.034602 5800 scope.go:122] "RemoveContainer" containerID="6db23322f50ea2917c7533c11fa9b0d0df449d5108b51e8bb6db454ded748b39" Nov 02 21:06:52.697796 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:06:52.697727 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:07:20.042876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:07:20.042835 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt"] Nov 02 21:07:20.089022 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:07:20.088982 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b3a6345-6w8pt"] Nov 02 21:07:21.623419 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:07:21.623385 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff3b757-8b28-4ae0-a030-5b5e3feef2bb" path="/var/lib/kubelet/pods/0ff3b757-8b28-4ae0-a030-5b5e3feef2bb/volumes" Nov 02 21:07:22.696973 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:07:22.696934 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:07:26.068742 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:07:26.068700 5800 scope.go:122] "RemoveContainer" containerID="1e6ba3ae2498fdacb057b6e759d5652fb18ab760f1e58bc32fcf14fd492853a3" Nov 02 21:07:26.083681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:07:26.083649 5800 scope.go:122] "RemoveContainer" containerID="88eccd9606044fb5361f21aa7535f64f38b3858690d64bf2414c0fb7cdcd8718" Nov 02 21:07:26.093712 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:07:26.093689 5800 scope.go:122] "RemoveContainer" containerID="c24ea7bb02721c45baa3ace77b1b00b1b13d0053718b22e5b4078b90bf920a2c" Nov 02 21:07:52.697009 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:07:52.696969 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:08:15.708573 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:15.708475 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv"] Nov 02 21:08:20.709606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:20.709562 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" podUID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" containerName="payload-pre-processing" containerID="cri-o://4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e" gracePeriod=25 Nov 02 21:08:22.545527 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:22.545492 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:08:22.549134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:22.549112 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:08:22.697823 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:08:22.697778 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:08:25.728907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:25.728863 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" podUID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:08:26.131841 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:26.131806 5800 scope.go:122] "RemoveContainer" containerID="aace42eea3a0b20a289a775d5bcff056fa36ad18af9c8c5d66f401e50a69090e" Nov 02 21:08:35.729820 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:35.729777 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" podUID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:08:41.008690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.008659 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 21:08:41.061097 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.061058 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-plugins-config-volume\") pod \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " Nov 02 21:08:41.061330 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.061276 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-kube-api-access-hqw4d\" (UniqueName: \"kubernetes.io/projected/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-kube-api-access-hqw4d\") pod \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\" (UID: \"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6\") " Nov 02 21:08:41.061506 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.061482 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-plugins-config-volume" pod "c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" (UID: "c0fbe41d-cf10-4b0d-85e0-cb318d4426f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:08:41.061814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.061794 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:41.063350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.063325 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-kube-api-access-hqw4d" pod "c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" (UID: "c0fbe41d-cf10-4b0d-85e0-cb318d4426f6"). InnerVolumeSpecName "kube-api-access-hqw4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:08:41.172989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.168230 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqw4d\" (UniqueName: \"kubernetes.io/projected/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6-kube-api-access-hqw4d\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:08:41.611695 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.611657 5800 generic.go:356] "Generic (PLEG): container finished" podID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" containerID="4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e" exitCode=0 Nov 02 21:08:41.611917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.611713 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" Nov 02 21:08:41.611917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.611747 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" event={"ID":"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6","Type":"ContainerDied","Data":"4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e"} Nov 02 21:08:41.611917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.611832 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv" event={"ID":"c0fbe41d-cf10-4b0d-85e0-cb318d4426f6","Type":"ContainerDied","Data":"109ac18dac459dbac5ce64b817e17c27c79360d7b95b5b088870750aec80a839"} Nov 02 21:08:41.611917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.611856 5800 scope.go:122] "RemoveContainer" containerID="4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e" Nov 02 21:08:41.624852 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.624817 5800 scope.go:122] "RemoveContainer" containerID="4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e" Nov 02 21:08:41.625098 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:08:41.625079 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e\": container with ID starting with 4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e not found: ID does not exist" containerID="4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e" Nov 02 21:08:41.625163 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.625105 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e"} err="failed to get container status \"4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e\": rpc error: code = NotFound desc = could not find container \"4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e\": container with ID starting with 4bce2c6c693901f78c895a0ec9421053bd2210bef50991056b43087f0a95311e not found: ID does not exist" Nov 02 21:08:41.766336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.766264 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv"] Nov 02 21:08:41.812879 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:41.812844 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2c6ce9-6bd75cffcf-qjnfv"] Nov 02 21:08:43.623090 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:43.623052 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" path="/var/lib/kubelet/pods/c0fbe41d-cf10-4b0d-85e0-cb318d4426f6/volumes" Nov 02 21:08:52.466297 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:52.466250 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb"] Nov 02 21:08:52.568345 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:52.568309 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q"] Nov 02 21:08:52.702334 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:08:52.702265 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:08:57.467145 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:57.467086 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" podUID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" containerName="payload-processing" containerID="cri-o://2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648" gracePeriod=25 Nov 02 21:08:57.568833 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:57.568781 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" podUID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" containerName="payload-pre-processing" containerID="cri-o://6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6" gracePeriod=25 Nov 02 21:08:58.011878 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:58.011836 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" podUID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:08:58.012111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:08:58.011838 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" podUID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:09:02.087646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:02.087611 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb"] Nov 02 21:09:02.134487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:02.134447 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55bfd721-zgcrb"] Nov 02 21:09:03.622719 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:03.622681 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58d2e8b-d78b-4b04-859e-c1fe73611ada" path="/var/lib/kubelet/pods/f58d2e8b-d78b-4b04-859e-c1fe73611ada/volumes" Nov 02 21:09:08.012158 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:08.012107 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" podUID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:09:08.012600 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:08.012159 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" podUID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:09:17.816798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.816757 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 21:09:17.881479 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.881450 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 21:09:17.896159 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.896128 5800 generic.go:356] "Generic (PLEG): container finished" podID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" containerID="2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648" exitCode=0 Nov 02 21:09:17.896370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.896175 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" Nov 02 21:09:17.896370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.896214 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" event={"ID":"a0f5768d-4731-4a0f-9701-5f35fbb6129f","Type":"ContainerDied","Data":"2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648"} Nov 02 21:09:17.896370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.896277 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb" event={"ID":"a0f5768d-4731-4a0f-9701-5f35fbb6129f","Type":"ContainerDied","Data":"e207dcca425e10f5a368a12dc92ffe98abbdc25dd362e7f51e9b0f8007e573cd"} Nov 02 21:09:17.896370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.896310 5800 scope.go:122] "RemoveContainer" containerID="2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648" Nov 02 21:09:17.897885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.897861 5800 generic.go:356] "Generic (PLEG): container finished" podID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" containerID="6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6" exitCode=0 Nov 02 21:09:17.897992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.897910 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" Nov 02 21:09:17.897992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.897924 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" event={"ID":"f7bbf1c0-99cf-4945-88fa-34fdda2d2633","Type":"ContainerDied","Data":"6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6"} Nov 02 21:09:17.897992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.897964 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q" event={"ID":"f7bbf1c0-99cf-4945-88fa-34fdda2d2633","Type":"ContainerDied","Data":"a4ec3c31f47506dba706b264012b50b03b34efd57331ec7ed93c151422910de7"} Nov 02 21:09:17.907511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.907486 5800 scope.go:122] "RemoveContainer" containerID="2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648" Nov 02 21:09:17.907880 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:09:17.907857 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648\": container with ID starting with 2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648 not found: ID does not exist" containerID="2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648" Nov 02 21:09:17.907959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.907889 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648"} err="failed to get container status \"2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648\": rpc error: code = NotFound desc = could not find container \"2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648\": container with ID starting with 2d02a27e06bfb232b5642e6e42e4a832497b3043109f232b12ed0a0cd67d6648 not found: ID does not exist" Nov 02 21:09:17.907959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.907907 5800 scope.go:122] "RemoveContainer" containerID="6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6" Nov 02 21:09:17.918197 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.918176 5800 scope.go:122] "RemoveContainer" containerID="6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6" Nov 02 21:09:17.918578 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:09:17.918553 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6\": container with ID starting with 6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6 not found: ID does not exist" containerID="6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6" Nov 02 21:09:17.918649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.918586 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6"} err="failed to get container status \"6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6\": rpc error: code = NotFound desc = could not find container \"6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6\": container with ID starting with 6c0ad8aee9e2583acc89e66f4f9544c97ed5eace947247e9b57106832ab6e2e6 not found: ID does not exist" Nov 02 21:09:17.966791 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.966748 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-kube-api-access-p478x\" (UniqueName: \"kubernetes.io/projected/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-kube-api-access-p478x\") pod \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " Nov 02 21:09:17.967008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.966908 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-plugins-config-volume\") pod \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\" (UID: \"f7bbf1c0-99cf-4945-88fa-34fdda2d2633\") " Nov 02 21:09:17.967097 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.967028 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f5768d-4731-4a0f-9701-5f35fbb6129f-kube-api-access-n6rmp\" (UniqueName: \"kubernetes.io/projected/a0f5768d-4731-4a0f-9701-5f35fbb6129f-kube-api-access-n6rmp\") pod \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " Nov 02 21:09:17.967097 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.967076 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f5768d-4731-4a0f-9701-5f35fbb6129f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f5768d-4731-4a0f-9701-5f35fbb6129f-plugins-config-volume\") pod \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\" (UID: \"a0f5768d-4731-4a0f-9701-5f35fbb6129f\") " Nov 02 21:09:17.967595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.967561 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-plugins-config-volume" pod "f7bbf1c0-99cf-4945-88fa-34fdda2d2633" (UID: "f7bbf1c0-99cf-4945-88fa-34fdda2d2633"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:17.967743 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.967681 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f5768d-4731-4a0f-9701-5f35fbb6129f-plugins-config-volume" pod "a0f5768d-4731-4a0f-9701-5f35fbb6129f" (UID: "a0f5768d-4731-4a0f-9701-5f35fbb6129f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:09:17.969439 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.969416 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f5768d-4731-4a0f-9701-5f35fbb6129f-kube-api-access-n6rmp" pod "a0f5768d-4731-4a0f-9701-5f35fbb6129f" (UID: "a0f5768d-4731-4a0f-9701-5f35fbb6129f"). InnerVolumeSpecName "kube-api-access-n6rmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:17.969811 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:17.969795 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-kube-api-access-p478x" pod "f7bbf1c0-99cf-4945-88fa-34fdda2d2633" (UID: "f7bbf1c0-99cf-4945-88fa-34fdda2d2633"). InnerVolumeSpecName "kube-api-access-p478x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:09:18.068799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.068708 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p478x\" (UniqueName: \"kubernetes.io/projected/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-kube-api-access-p478x\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:18.068799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.068738 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bbf1c0-99cf-4945-88fa-34fdda2d2633-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:18.068799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.068749 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6rmp\" (UniqueName: \"kubernetes.io/projected/a0f5768d-4731-4a0f-9701-5f35fbb6129f-kube-api-access-n6rmp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:18.068799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.068758 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0f5768d-4731-4a0f-9701-5f35fbb6129f-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:09:18.314597 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.314564 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q"] Nov 02 21:09:18.317398 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.317370 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-78bef7-54d69594c5-7pt8q"] Nov 02 21:09:18.418207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.418105 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb"] Nov 02 21:09:18.422559 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:18.422525 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-78bef7-5559cc48df-rkdcb"] Nov 02 21:09:19.627415 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:19.627373 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" path="/var/lib/kubelet/pods/a0f5768d-4731-4a0f-9701-5f35fbb6129f/volumes" Nov 02 21:09:19.628111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:19.628088 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" path="/var/lib/kubelet/pods/f7bbf1c0-99cf-4945-88fa-34fdda2d2633/volumes" Nov 02 21:09:22.697221 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:09:22.697180 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:09:52.696870 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:09:52.696833 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:09:56.939410 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:09:56.939369 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt"] Nov 02 21:10:01.940335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:01.940251 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" podUID="d86154ed-afba-4f0b-a598-a10a3dd9003e" containerName="payload-processing" containerID="cri-o://c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274" gracePeriod=25 Nov 02 21:10:02.236676 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.236645 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:10:02.244824 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.244787 5800 generic.go:356] "Generic (PLEG): container finished" podID="d86154ed-afba-4f0b-a598-a10a3dd9003e" containerID="c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274" exitCode=0 Nov 02 21:10:02.245017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.244836 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" Nov 02 21:10:02.245017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.244872 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" event={"ID":"d86154ed-afba-4f0b-a598-a10a3dd9003e","Type":"ContainerDied","Data":"c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274"} Nov 02 21:10:02.245017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.244977 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt" event={"ID":"d86154ed-afba-4f0b-a598-a10a3dd9003e","Type":"ContainerDied","Data":"5054b2aa3f9ad1037e95a238749df30c6bb303741c96075204a8d83540fed1ac"} Nov 02 21:10:02.245017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.245012 5800 scope.go:122] "RemoveContainer" containerID="c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274" Nov 02 21:10:02.258000 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.257911 5800 scope.go:122] "RemoveContainer" containerID="c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274" Nov 02 21:10:02.258385 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:02.258354 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274\": container with ID starting with c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274 not found: ID does not exist" containerID="c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274" Nov 02 21:10:02.258491 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.258397 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274"} err="failed to get container status \"c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274\": rpc error: code = NotFound desc = could not find container \"c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274\": container with ID starting with c22c9513cc641896e105df88d03130faff12ff5350fd1cbc7fbb9dfc2f55d274 not found: ID does not exist" Nov 02 21:10:02.361244 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.361203 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86154ed-afba-4f0b-a598-a10a3dd9003e-kube-api-access-6fw8t\" (UniqueName: \"kubernetes.io/projected/d86154ed-afba-4f0b-a598-a10a3dd9003e-kube-api-access-6fw8t\") pod \"d86154ed-afba-4f0b-a598-a10a3dd9003e\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " Nov 02 21:10:02.361485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.361342 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86154ed-afba-4f0b-a598-a10a3dd9003e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86154ed-afba-4f0b-a598-a10a3dd9003e-plugins-config-volume\") pod \"d86154ed-afba-4f0b-a598-a10a3dd9003e\" (UID: \"d86154ed-afba-4f0b-a598-a10a3dd9003e\") " Nov 02 21:10:02.361821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.361792 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86154ed-afba-4f0b-a598-a10a3dd9003e-plugins-config-volume" pod "d86154ed-afba-4f0b-a598-a10a3dd9003e" (UID: "d86154ed-afba-4f0b-a598-a10a3dd9003e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:02.361975 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.361952 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d86154ed-afba-4f0b-a598-a10a3dd9003e-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:02.363427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.363396 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86154ed-afba-4f0b-a598-a10a3dd9003e-kube-api-access-6fw8t" pod "d86154ed-afba-4f0b-a598-a10a3dd9003e" (UID: "d86154ed-afba-4f0b-a598-a10a3dd9003e"). InnerVolumeSpecName "kube-api-access-6fw8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:02.463727 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.463633 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fw8t\" (UniqueName: \"kubernetes.io/projected/d86154ed-afba-4f0b-a598-a10a3dd9003e-kube-api-access-6fw8t\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:02.660757 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.660713 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt"] Nov 02 21:10:02.665222 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:02.665190 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68a6c923-56bcbb7847-g4blt"] Nov 02 21:10:03.624018 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:03.623983 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86154ed-afba-4f0b-a598-a10a3dd9003e" path="/var/lib/kubelet/pods/d86154ed-afba-4f0b-a598-a10a3dd9003e/volumes" Nov 02 21:10:05.332768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:05.332724 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh"] Nov 02 21:10:05.333445 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:05.333412 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" containerID="cri-o://6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702" gracePeriod=30 Nov 02 21:10:06.466418 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.466384 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg"] Nov 02 21:10:06.467015 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467000 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" containerName="payload-pre-processing" Nov 02 21:10:06.467062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467016 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" containerName="payload-pre-processing" Nov 02 21:10:06.467062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467031 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" containerName="payload-processing" Nov 02 21:10:06.467062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467036 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" containerName="payload-processing" Nov 02 21:10:06.467062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467058 5800 cpu_manager.go:414] "containerMap: removing container" podUID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" containerName="payload-pre-processing" Nov 02 21:10:06.467209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467064 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" containerName="payload-pre-processing" Nov 02 21:10:06.467209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467094 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d86154ed-afba-4f0b-a598-a10a3dd9003e" containerName="payload-processing" Nov 02 21:10:06.467209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467099 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86154ed-afba-4f0b-a598-a10a3dd9003e" containerName="payload-processing" Nov 02 21:10:06.467209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467187 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86154ed-afba-4f0b-a598-a10a3dd9003e" containerName="payload-processing" Nov 02 21:10:06.467209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467196 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fbe41d-cf10-4b0d-85e0-cb318d4426f6" containerName="payload-pre-processing" Nov 02 21:10:06.467209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467203 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f5768d-4731-4a0f-9701-5f35fbb6129f" containerName="payload-processing" Nov 02 21:10:06.467430 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.467212 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bbf1c0-99cf-4945-88fa-34fdda2d2633" containerName="payload-pre-processing" Nov 02 21:10:06.472541 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.472516 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.523386 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.523351 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg"] Nov 02 21:10:06.564689 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.564651 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:10:06.564876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.564697 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:10:06.611250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.611217 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:10:06.611250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.611217 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:10:06.612814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.612793 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704b6373-074e-4990-862c-cb99972eefdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.612871 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.612834 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4mx\" (UniqueName: \"kubernetes.io/projected/704b6373-074e-4990-862c-cb99972eefdd-kube-api-access-8p4mx\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.613019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.613000 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704b6373-074e-4990-862c-cb99972eefdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.713727 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.713690 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704b6373-074e-4990-862c-cb99972eefdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.713946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.713741 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4mx\" (UniqueName: \"kubernetes.io/projected/704b6373-074e-4990-862c-cb99972eefdd-kube-api-access-8p4mx\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.713946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.713850 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704b6373-074e-4990-862c-cb99972eefdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.714515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.714487 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704b6373-074e-4990-862c-cb99972eefdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.716203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.716180 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704b6373-074e-4990-862c-cb99972eefdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.766683 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.766647 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4mx\" (UniqueName: \"kubernetes.io/projected/704b6373-074e-4990-862c-cb99972eefdd-kube-api-access-8p4mx\") pod \"maas-api-revoke-sub-6439a46e-rlmsg\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:06.786363 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:06.786336 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:07.026380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:07.026271 5800 patch_prober.go:29] interesting pod/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:10:07.026380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:07.026348 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:10:07.158332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:07.158298 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg"] Nov 02 21:10:07.159987 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:07.159960 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704b6373_074e_4990_862c_cb99972eefdd.slice/crio-89d011c665b65c5d2c98fccfa3384883751a97eb2af1ba8c0d9ac2f766a680d9 WatchSource:0}: Error finding container 89d011c665b65c5d2c98fccfa3384883751a97eb2af1ba8c0d9ac2f766a680d9: Status 404 returned error can't find the container with id 89d011c665b65c5d2c98fccfa3384883751a97eb2af1ba8c0d9ac2f766a680d9 Nov 02 21:10:07.289572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:07.289490 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" event={"ID":"704b6373-074e-4990-862c-cb99972eefdd","Type":"ContainerStarted","Data":"89d011c665b65c5d2c98fccfa3384883751a97eb2af1ba8c0d9ac2f766a680d9"} Nov 02 21:10:08.329447 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:08.329356 5800 generic.go:356] "Generic (PLEG): container finished" podID="704b6373-074e-4990-862c-cb99972eefdd" containerID="07ce150ca5eeccb1125bf3e8b35a501d483e0838f19e3b7e6a440a1717f2ef98" exitCode=0 Nov 02 21:10:08.330087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:08.329511 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" event={"ID":"704b6373-074e-4990-862c-cb99972eefdd","Type":"ContainerDied","Data":"07ce150ca5eeccb1125bf3e8b35a501d483e0838f19e3b7e6a440a1717f2ef98"} Nov 02 21:10:09.499524 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.499485 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247"] Nov 02 21:10:09.513722 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.513701 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.534893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.534861 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:09.555663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.555634 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247"] Nov 02 21:10:09.558603 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.558576 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704b6373-074e-4990-862c-cb99972eefdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704b6373-074e-4990-862c-cb99972eefdd-maas-api-openshift-ca\") pod \"704b6373-074e-4990-862c-cb99972eefdd\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " Nov 02 21:10:09.558747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.558632 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/704b6373-074e-4990-862c-cb99972eefdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704b6373-074e-4990-862c-cb99972eefdd-maas-api-xks-ca\") pod \"704b6373-074e-4990-862c-cb99972eefdd\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " Nov 02 21:10:09.558747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.558716 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704b6373-074e-4990-862c-cb99972eefdd-kube-api-access-8p4mx\" (UniqueName: \"kubernetes.io/projected/704b6373-074e-4990-862c-cb99972eefdd-kube-api-access-8p4mx\") pod \"704b6373-074e-4990-862c-cb99972eefdd\" (UID: \"704b6373-074e-4990-862c-cb99972eefdd\") " Nov 02 21:10:09.559004 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.558980 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704b6373-074e-4990-862c-cb99972eefdd-maas-api-openshift-ca" pod "704b6373-074e-4990-862c-cb99972eefdd" (UID: "704b6373-074e-4990-862c-cb99972eefdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:09.559089 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.559073 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.559162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.559142 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.559340 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.559270 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4849\" (UniqueName: \"kubernetes.io/projected/9c97d57e-103a-4f15-b12e-2579cc0e1b88-kube-api-access-h4849\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.559464 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.559422 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704b6373-074e-4990-862c-cb99972eefdd-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:09.561392 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.561365 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704b6373-074e-4990-862c-cb99972eefdd-maas-api-xks-ca" pod "704b6373-074e-4990-862c-cb99972eefdd" (UID: "704b6373-074e-4990-862c-cb99972eefdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:09.562444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.562416 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704b6373-074e-4990-862c-cb99972eefdd-kube-api-access-8p4mx" pod "704b6373-074e-4990-862c-cb99972eefdd" (UID: "704b6373-074e-4990-862c-cb99972eefdd"). InnerVolumeSpecName "kube-api-access-8p4mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:09.660276 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.660238 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.660505 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.660350 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.660505 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.660430 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4849\" (UniqueName: \"kubernetes.io/projected/9c97d57e-103a-4f15-b12e-2579cc0e1b88-kube-api-access-h4849\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.660628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.660587 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704b6373-074e-4990-862c-cb99972eefdd-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:09.660628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.660604 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4mx\" (UniqueName: \"kubernetes.io/projected/704b6373-074e-4990-862c-cb99972eefdd-kube-api-access-8p4mx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:09.661102 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.661078 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.663259 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.663230 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.716219 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.716177 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4849\" (UniqueName: \"kubernetes.io/projected/9c97d57e-103a-4f15-b12e-2579cc0e1b88-kube-api-access-h4849\") pod \"maas-api-revoke-sub-f04e5278-fr247\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:09.845303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:09.844636 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:10.219617 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.219587 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247"] Nov 02 21:10:10.220464 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:10.220435 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c97d57e_103a_4f15_b12e_2579cc0e1b88.slice/crio-b964d1080a6c6293c16a4f30bb28310cc8ec8015c53aa35fe8e8d45dd89793ae WatchSource:0}: Error finding container b964d1080a6c6293c16a4f30bb28310cc8ec8015c53aa35fe8e8d45dd89793ae: Status 404 returned error can't find the container with id b964d1080a6c6293c16a4f30bb28310cc8ec8015c53aa35fe8e8d45dd89793ae Nov 02 21:10:10.346715 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.346678 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" event={"ID":"9c97d57e-103a-4f15-b12e-2579cc0e1b88","Type":"ContainerStarted","Data":"b964d1080a6c6293c16a4f30bb28310cc8ec8015c53aa35fe8e8d45dd89793ae"} Nov 02 21:10:10.348174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.348145 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" Nov 02 21:10:10.348326 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.348176 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg" event={"ID":"704b6373-074e-4990-862c-cb99972eefdd","Type":"ContainerDied","Data":"89d011c665b65c5d2c98fccfa3384883751a97eb2af1ba8c0d9ac2f766a680d9"} Nov 02 21:10:10.348326 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.348223 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89d011c665b65c5d2c98fccfa3384883751a97eb2af1ba8c0d9ac2f766a680d9" Nov 02 21:10:10.615778 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.615747 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 21:10:10.673522 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673483 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-istio-token\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-istio-token\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.673705 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673536 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-credential-socket\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.673705 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673674 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-certs\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.673798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673723 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-kube-api-access-ncq6z\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-kube-api-access-ncq6z\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.673798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673758 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-envoy\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.673908 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673866 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-credential-socket" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:10.673977 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673938 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22126ebf-024b-45aa-91d1-50488241b3ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22126ebf-024b-45aa-91d1-50488241b3ba-istiod-ca-cert\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.674035 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674023 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22126ebf-024b-45aa-91d1-50488241b3ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22126ebf-024b-45aa-91d1-50488241b3ba-istio-podinfo\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.674099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.673940 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-certs" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:10.674165 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674109 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-data\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.674165 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674149 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-socket\") pod \"22126ebf-024b-45aa-91d1-50488241b3ba\" (UID: \"22126ebf-024b-45aa-91d1-50488241b3ba\") " Nov 02 21:10:10.674324 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674268 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22126ebf-024b-45aa-91d1-50488241b3ba-istiod-ca-cert" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:10.674518 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674500 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-socket" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:10.674660 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674637 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-data" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:10.674930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674910 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.675019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674935 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.675019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674951 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.675019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674965 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.675019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.674979 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22126ebf-024b-45aa-91d1-50488241b3ba-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.676063 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.676041 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-kube-api-access-ncq6z" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "kube-api-access-ncq6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:10.676203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.676186 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-istio-token" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:10.676312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.676272 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-envoy" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:10.677027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.677003 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22126ebf-024b-45aa-91d1-50488241b3ba-istio-podinfo" pod "22126ebf-024b-45aa-91d1-50488241b3ba" (UID: "22126ebf-024b-45aa-91d1-50488241b3ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:10:10.755410 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.755375 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5"] Nov 02 21:10:10.776135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.776057 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22126ebf-024b-45aa-91d1-50488241b3ba-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.776135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.776095 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.776135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.776110 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncq6z\" (UniqueName: \"kubernetes.io/projected/22126ebf-024b-45aa-91d1-50488241b3ba-kube-api-access-ncq6z\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.776135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.776126 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22126ebf-024b-45aa-91d1-50488241b3ba-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:10.780450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:10.780413 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj"] Nov 02 21:10:11.357643 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.357606 5800 generic.go:356] "Generic (PLEG): container finished" podID="9c97d57e-103a-4f15-b12e-2579cc0e1b88" containerID="0225101bd754208b0224fbd5c0c866a093c4a730d6ea8a878f946b7e93d7a793" exitCode=0 Nov 02 21:10:11.357868 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.357691 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" event={"ID":"9c97d57e-103a-4f15-b12e-2579cc0e1b88","Type":"ContainerDied","Data":"0225101bd754208b0224fbd5c0c866a093c4a730d6ea8a878f946b7e93d7a793"} Nov 02 21:10:11.359853 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.359819 5800 generic.go:356] "Generic (PLEG): container finished" podID="22126ebf-024b-45aa-91d1-50488241b3ba" containerID="6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702" exitCode=0 Nov 02 21:10:11.359997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.359879 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" Nov 02 21:10:11.359997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.359894 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" event={"ID":"22126ebf-024b-45aa-91d1-50488241b3ba","Type":"ContainerDied","Data":"6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702"} Nov 02 21:10:11.359997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.359938 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh" event={"ID":"22126ebf-024b-45aa-91d1-50488241b3ba","Type":"ContainerDied","Data":"80bacf54d08f44d1cf2987ca13a43877fce54fa2dab30c980bb936d51145296a"} Nov 02 21:10:11.359997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.359959 5800 scope.go:122] "RemoveContainer" containerID="6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702" Nov 02 21:10:11.371613 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.371592 5800 scope.go:122] "RemoveContainer" containerID="cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e" Nov 02 21:10:11.381981 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.381962 5800 scope.go:122] "RemoveContainer" containerID="6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702" Nov 02 21:10:11.382262 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:11.382246 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702\": container with ID starting with 6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702 not found: ID does not exist" containerID="6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702" Nov 02 21:10:11.382332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.382270 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702"} err="failed to get container status \"6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702\": rpc error: code = NotFound desc = could not find container \"6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702\": container with ID starting with 6b12112483f769683c3e344a3c949eaca1f10af023c47bcca950649e07ceb702 not found: ID does not exist" Nov 02 21:10:11.382332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.382307 5800 scope.go:122] "RemoveContainer" containerID="cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e" Nov 02 21:10:11.382529 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:11.382513 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e\": container with ID starting with cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e not found: ID does not exist" containerID="cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e" Nov 02 21:10:11.382570 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.382535 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e"} err="failed to get container status \"cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e\": rpc error: code = NotFound desc = could not find container \"cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e\": container with ID starting with cbc2c7a2f685f80c6623445abf3f7efea8da81c5a7aae552c1ea37570b81005e not found: ID does not exist" Nov 02 21:10:11.571233 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.571194 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh"] Nov 02 21:10:11.574243 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.574214 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-78bef7-openshift-default-68879c6b87-slgbh"] Nov 02 21:10:11.623789 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:11.623756 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" path="/var/lib/kubelet/pods/22126ebf-024b-45aa-91d1-50488241b3ba/volumes" Nov 02 21:10:12.543531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.543505 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:12.594583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.594551 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-xks-ca\") pod \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " Nov 02 21:10:12.594771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.594642 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-openshift-ca\") pod \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " Nov 02 21:10:12.594771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.594714 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c97d57e-103a-4f15-b12e-2579cc0e1b88-kube-api-access-h4849\" (UniqueName: \"kubernetes.io/projected/9c97d57e-103a-4f15-b12e-2579cc0e1b88-kube-api-access-h4849\") pod \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\" (UID: \"9c97d57e-103a-4f15-b12e-2579cc0e1b88\") " Nov 02 21:10:12.595037 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.595011 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-openshift-ca" pod "9c97d57e-103a-4f15-b12e-2579cc0e1b88" (UID: "9c97d57e-103a-4f15-b12e-2579cc0e1b88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:12.595216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.595195 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:12.596987 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.596959 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-xks-ca" pod "9c97d57e-103a-4f15-b12e-2579cc0e1b88" (UID: "9c97d57e-103a-4f15-b12e-2579cc0e1b88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:12.597508 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.597489 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c97d57e-103a-4f15-b12e-2579cc0e1b88-kube-api-access-h4849" pod "9c97d57e-103a-4f15-b12e-2579cc0e1b88" (UID: "9c97d57e-103a-4f15-b12e-2579cc0e1b88"). InnerVolumeSpecName "kube-api-access-h4849". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:12.696912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.696831 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97d57e-103a-4f15-b12e-2579cc0e1b88-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:12.696912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:12.696870 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4849\" (UniqueName: \"kubernetes.io/projected/9c97d57e-103a-4f15-b12e-2579cc0e1b88-kube-api-access-h4849\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:13.382104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:13.382073 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" Nov 02 21:10:13.382344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:13.382098 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247" event={"ID":"9c97d57e-103a-4f15-b12e-2579cc0e1b88","Type":"ContainerDied","Data":"b964d1080a6c6293c16a4f30bb28310cc8ec8015c53aa35fe8e8d45dd89793ae"} Nov 02 21:10:13.382344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:13.382160 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b964d1080a6c6293c16a4f30bb28310cc8ec8015c53aa35fe8e8d45dd89793ae" Nov 02 21:10:18.442822 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.442785 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf"] Nov 02 21:10:18.443392 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443377 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9c97d57e-103a-4f15-b12e-2579cc0e1b88" containerName="revoke-keys" Nov 02 21:10:18.443392 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443392 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c97d57e-103a-4f15-b12e-2579cc0e1b88" containerName="revoke-keys" Nov 02 21:10:18.443536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443417 5800 cpu_manager.go:414] "containerMap: removing container" podUID="704b6373-074e-4990-862c-cb99972eefdd" containerName="revoke-keys" Nov 02 21:10:18.443536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443422 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704b6373-074e-4990-862c-cb99972eefdd" containerName="revoke-keys" Nov 02 21:10:18.443536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443442 5800 cpu_manager.go:414] "containerMap: removing container" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" Nov 02 21:10:18.443536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443448 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" Nov 02 21:10:18.443536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443465 5800 cpu_manager.go:414] "containerMap: removing container" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" Nov 02 21:10:18.443536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443470 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" Nov 02 21:10:18.443846 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443547 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c97d57e-103a-4f15-b12e-2579cc0e1b88" containerName="revoke-keys" Nov 02 21:10:18.443846 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443560 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="704b6373-074e-4990-862c-cb99972eefdd" containerName="revoke-keys" Nov 02 21:10:18.443846 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443569 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" Nov 02 21:10:18.443846 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.443576 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="22126ebf-024b-45aa-91d1-50488241b3ba" containerName="istio-proxy" Nov 02 21:10:18.447220 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.447198 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.470126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.469316 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.470126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.469399 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl4tx\" (UniqueName: \"kubernetes.io/projected/81c2076a-a396-4088-ae79-9df49e9aaa42-kube-api-access-kl4tx\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.470126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.469495 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.496994 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.496960 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:10:18.500228 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.500197 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf"] Nov 02 21:10:18.542221 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.542181 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:10:18.542221 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.542181 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:10:18.542509 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.542314 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:10:18.570694 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.570660 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl4tx\" (UniqueName: \"kubernetes.io/projected/81c2076a-a396-4088-ae79-9df49e9aaa42-kube-api-access-kl4tx\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.570917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.570732 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.570917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.570870 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.571612 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.571591 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.573219 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.573190 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.623975 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.623942 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl4tx\" (UniqueName: \"kubernetes.io/projected/81c2076a-a396-4088-ae79-9df49e9aaa42-kube-api-access-kl4tx\") pod \"maas-api-revoke-sub-29913697-xj2zf\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:18.772529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:18.772484 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:19.149625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:19.149596 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf"] Nov 02 21:10:19.152125 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:19.152090 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c2076a_a396_4088_ae79_9df49e9aaa42.slice/crio-76fef85a9f9cba1c1fb35520a6aa4284985b7cdf5fbbfadcf85896e99623b9ac WatchSource:0}: Error finding container 76fef85a9f9cba1c1fb35520a6aa4284985b7cdf5fbbfadcf85896e99623b9ac: Status 404 returned error can't find the container with id 76fef85a9f9cba1c1fb35520a6aa4284985b7cdf5fbbfadcf85896e99623b9ac Nov 02 21:10:19.324033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:19.323997 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9"] Nov 02 21:10:19.423116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:19.423015 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" event={"ID":"81c2076a-a396-4088-ae79-9df49e9aaa42","Type":"ContainerStarted","Data":"76fef85a9f9cba1c1fb35520a6aa4284985b7cdf5fbbfadcf85896e99623b9ac"} Nov 02 21:10:20.284944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.284915 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb"] Nov 02 21:10:20.289472 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.289448 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.317833 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.317803 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb"] Nov 02 21:10:20.395040 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.394996 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.395216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.395184 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.395259 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.395220 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jdj\" (UniqueName: \"kubernetes.io/projected/9c86780e-3beb-4412-bc0e-153520c8eaf7-kube-api-access-58jdj\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.432382 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.432343 5800 generic.go:356] "Generic (PLEG): container finished" podID="81c2076a-a396-4088-ae79-9df49e9aaa42" containerID="7f6fbe3e0f82dae2310603b24bf78ffe96cb2d8b220d33cf01d40d047a05ea21" exitCode=0 Nov 02 21:10:20.432572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.432435 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" event={"ID":"81c2076a-a396-4088-ae79-9df49e9aaa42","Type":"ContainerDied","Data":"7f6fbe3e0f82dae2310603b24bf78ffe96cb2d8b220d33cf01d40d047a05ea21"} Nov 02 21:10:20.496313 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.496261 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.496495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.496407 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.496495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.496437 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jdj\" (UniqueName: \"kubernetes.io/projected/9c86780e-3beb-4412-bc0e-153520c8eaf7-kube-api-access-58jdj\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.497180 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.497154 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.503007 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.502975 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.554036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.553961 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jdj\" (UniqueName: \"kubernetes.io/projected/9c86780e-3beb-4412-bc0e-153520c8eaf7-kube-api-access-58jdj\") pod \"maas-api-revoke-sub-74c49ea1-vhrwb\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:20.660063 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:20.659584 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:21.354798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.354768 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb"] Nov 02 21:10:21.357053 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:21.357013 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c86780e_3beb_4412_bc0e_153520c8eaf7.slice/crio-01fb5a23414cee469dbe77283888f744defc882145452617b365dc8e4c71d484 WatchSource:0}: Error finding container 01fb5a23414cee469dbe77283888f744defc882145452617b365dc8e4c71d484: Status 404 returned error can't find the container with id 01fb5a23414cee469dbe77283888f744defc882145452617b365dc8e4c71d484 Nov 02 21:10:21.441896 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.441857 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" event={"ID":"9c86780e-3beb-4412-bc0e-153520c8eaf7","Type":"ContainerStarted","Data":"01fb5a23414cee469dbe77283888f744defc882145452617b365dc8e4c71d484"} Nov 02 21:10:21.595361 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.595333 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:21.634625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.634551 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-openshift-ca\") pod \"81c2076a-a396-4088-ae79-9df49e9aaa42\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " Nov 02 21:10:21.634798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.634627 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c2076a-a396-4088-ae79-9df49e9aaa42-kube-api-access-kl4tx\" (UniqueName: \"kubernetes.io/projected/81c2076a-a396-4088-ae79-9df49e9aaa42-kube-api-access-kl4tx\") pod \"81c2076a-a396-4088-ae79-9df49e9aaa42\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " Nov 02 21:10:21.634798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.634662 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-xks-ca\") pod \"81c2076a-a396-4088-ae79-9df49e9aaa42\" (UID: \"81c2076a-a396-4088-ae79-9df49e9aaa42\") " Nov 02 21:10:21.634967 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.634934 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-openshift-ca" pod "81c2076a-a396-4088-ae79-9df49e9aaa42" (UID: "81c2076a-a396-4088-ae79-9df49e9aaa42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:21.635131 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.635114 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:21.636821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.636798 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-xks-ca" pod "81c2076a-a396-4088-ae79-9df49e9aaa42" (UID: "81c2076a-a396-4088-ae79-9df49e9aaa42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:21.637202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.637187 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c2076a-a396-4088-ae79-9df49e9aaa42-kube-api-access-kl4tx" pod "81c2076a-a396-4088-ae79-9df49e9aaa42" (UID: "81c2076a-a396-4088-ae79-9df49e9aaa42"). InnerVolumeSpecName "kube-api-access-kl4tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:21.736096 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.736063 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl4tx\" (UniqueName: \"kubernetes.io/projected/81c2076a-a396-4088-ae79-9df49e9aaa42-kube-api-access-kl4tx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:21.736236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:21.736101 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c2076a-a396-4088-ae79-9df49e9aaa42-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:22.462923 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:22.462884 5800 generic.go:356] "Generic (PLEG): container finished" podID="9c86780e-3beb-4412-bc0e-153520c8eaf7" containerID="08fc42ac3f965e0825d90983bbd3aa31706c2ade5b21ffa75359fbd34bcab5c8" exitCode=0 Nov 02 21:10:22.463392 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:22.462965 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" event={"ID":"9c86780e-3beb-4412-bc0e-153520c8eaf7","Type":"ContainerDied","Data":"08fc42ac3f965e0825d90983bbd3aa31706c2ade5b21ffa75359fbd34bcab5c8"} Nov 02 21:10:22.464445 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:22.464420 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" Nov 02 21:10:22.464445 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:22.464440 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf" event={"ID":"81c2076a-a396-4088-ae79-9df49e9aaa42","Type":"ContainerDied","Data":"76fef85a9f9cba1c1fb35520a6aa4284985b7cdf5fbbfadcf85896e99623b9ac"} Nov 02 21:10:22.464656 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:22.464475 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76fef85a9f9cba1c1fb35520a6aa4284985b7cdf5fbbfadcf85896e99623b9ac" Nov 02 21:10:22.697486 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:22.697395 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:10:23.658317 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.651670 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:23.677260 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.665090 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c86780e-3beb-4412-bc0e-153520c8eaf7-kube-api-access-58jdj\" (UniqueName: \"kubernetes.io/projected/9c86780e-3beb-4412-bc0e-153520c8eaf7-kube-api-access-58jdj\") pod \"9c86780e-3beb-4412-bc0e-153520c8eaf7\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " Nov 02 21:10:23.678235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.678189 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-xks-ca\") pod \"9c86780e-3beb-4412-bc0e-153520c8eaf7\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " Nov 02 21:10:23.678435 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.678316 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-openshift-ca\") pod \"9c86780e-3beb-4412-bc0e-153520c8eaf7\" (UID: \"9c86780e-3beb-4412-bc0e-153520c8eaf7\") " Nov 02 21:10:23.691304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.680042 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-openshift-ca" pod "9c86780e-3beb-4412-bc0e-153520c8eaf7" (UID: "9c86780e-3beb-4412-bc0e-153520c8eaf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:23.705306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.692298 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-xks-ca" pod "9c86780e-3beb-4412-bc0e-153520c8eaf7" (UID: "9c86780e-3beb-4412-bc0e-153520c8eaf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:23.705306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.692727 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c86780e-3beb-4412-bc0e-153520c8eaf7-kube-api-access-58jdj" pod "9c86780e-3beb-4412-bc0e-153520c8eaf7" (UID: "9c86780e-3beb-4412-bc0e-153520c8eaf7"). InnerVolumeSpecName "kube-api-access-58jdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:23.780122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.780008 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jdj\" (UniqueName: \"kubernetes.io/projected/9c86780e-3beb-4412-bc0e-153520c8eaf7-kube-api-access-58jdj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:23.780353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.780338 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:23.780438 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:23.780429 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c86780e-3beb-4412-bc0e-153520c8eaf7-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:24.324883 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:24.324812 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" podUID="83840101-e53a-436f-ae00-bcdd34063062" containerName="payload-processing" containerID="cri-o://ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4" gracePeriod=25 Nov 02 21:10:24.481557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:24.481523 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" Nov 02 21:10:24.481745 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:24.481564 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb" event={"ID":"9c86780e-3beb-4412-bc0e-153520c8eaf7","Type":"ContainerDied","Data":"01fb5a23414cee469dbe77283888f744defc882145452617b365dc8e4c71d484"} Nov 02 21:10:24.481745 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:24.481617 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01fb5a23414cee469dbe77283888f744defc882145452617b365dc8e4c71d484" Nov 02 21:10:25.748154 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.748113 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq"] Nov 02 21:10:25.749094 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.749069 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9c86780e-3beb-4412-bc0e-153520c8eaf7" containerName="revoke-keys" Nov 02 21:10:25.749204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.749097 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c86780e-3beb-4412-bc0e-153520c8eaf7" containerName="revoke-keys" Nov 02 21:10:25.749204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.749142 5800 cpu_manager.go:414] "containerMap: removing container" podUID="81c2076a-a396-4088-ae79-9df49e9aaa42" containerName="revoke-keys" Nov 02 21:10:25.749204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.749150 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c2076a-a396-4088-ae79-9df49e9aaa42" containerName="revoke-keys" Nov 02 21:10:25.749341 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.749327 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c86780e-3beb-4412-bc0e-153520c8eaf7" containerName="revoke-keys" Nov 02 21:10:25.749381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.749352 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c2076a-a396-4088-ae79-9df49e9aaa42" containerName="revoke-keys" Nov 02 21:10:25.753469 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.753447 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.779532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.779487 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" containerID="cri-o://399abbd08706a5245015ada21011cd7b0b212f5b7b7fe46c6aa9d94d15ad962b" gracePeriod=45 Nov 02 21:10:25.801933 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.801893 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" containerID="cri-o://2bca9f0668540e5ada85a82313071845b2fdeb71d48e55dedcaef119a166fca4" gracePeriod=45 Nov 02 21:10:25.802574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.802174 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:10:25.802574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.802227 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:10:25.802574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.802244 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:10:25.802574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.802305 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq"] Nov 02 21:10:25.816465 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.816435 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtj8r\" (UniqueName: \"kubernetes.io/projected/536a6fca-b3a8-4fc7-b212-8606774bfd24-kube-api-access-rtj8r\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.816666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.816562 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.816666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.816649 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.844210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.844176 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:10:25.919255 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.919218 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.919441 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.919333 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.919511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.919490 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtj8r\" (UniqueName: \"kubernetes.io/projected/536a6fca-b3a8-4fc7-b212-8606774bfd24-kube-api-access-rtj8r\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.920312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.920272 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.921861 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.921838 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:25.975884 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:25.975850 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtj8r\" (UniqueName: \"kubernetes.io/projected/536a6fca-b3a8-4fc7-b212-8606774bfd24-kube-api-access-rtj8r\") pod \"maas-api-revoke-sub-37b85e70-575dq\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:26.069683 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.067988 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:26.096227 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.096201 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 21:10:26.122381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.122353 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 21:10:26.199725 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.199690 5800 scope.go:122] "RemoveContainer" containerID="6b5cdd89b1baffb8a1f446033e418bc3c05c8978416646f938cbd08045d3bdef" Nov 02 21:10:26.212354 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.212327 5800 scope.go:122] "RemoveContainer" containerID="399abbd08706a5245015ada21011cd7b0b212f5b7b7fe46c6aa9d94d15ad962b" Nov 02 21:10:26.223620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.223591 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0435af2-55cd-4cc3-b129-22a290db98ed-kube-api-access-rpq44\" (UniqueName: \"kubernetes.io/projected/c0435af2-55cd-4cc3-b129-22a290db98ed-kube-api-access-rpq44\") pod \"c0435af2-55cd-4cc3-b129-22a290db98ed\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " Nov 02 21:10:26.223771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.223697 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-home\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-home\") pod \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " Nov 02 21:10:26.223959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.223930 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-dshm\") pod \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " Nov 02 21:10:26.224091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.223975 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-home" pod "bf273e63-6dd2-49ea-9ffe-dca9a00de929" (UID: "bf273e63-6dd2-49ea-9ffe-dca9a00de929"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.224091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224003 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-model-cache\") pod \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " Nov 02 21:10:26.224091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224043 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-model-cache\") pod \"c0435af2-55cd-4cc3-b129-22a290db98ed\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " Nov 02 21:10:26.224091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224089 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tls-certs\") pod \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " Nov 02 21:10:26.224384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224125 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-home\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-home\") pod \"c0435af2-55cd-4cc3-b129-22a290db98ed\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " Nov 02 21:10:26.224384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224174 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-dshm\") pod \"c0435af2-55cd-4cc3-b129-22a290db98ed\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " Nov 02 21:10:26.224384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224213 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-tmp-dir\") pod \"c0435af2-55cd-4cc3-b129-22a290db98ed\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " Nov 02 21:10:26.224384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224265 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0435af2-55cd-4cc3-b129-22a290db98ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0435af2-55cd-4cc3-b129-22a290db98ed-tls-certs\") pod \"c0435af2-55cd-4cc3-b129-22a290db98ed\" (UID: \"c0435af2-55cd-4cc3-b129-22a290db98ed\") " Nov 02 21:10:26.224384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224338 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tmp-dir\") pod \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " Nov 02 21:10:26.224686 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224420 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf273e63-6dd2-49ea-9ffe-dca9a00de929-kube-api-access-dqttc\" (UniqueName: \"kubernetes.io/projected/bf273e63-6dd2-49ea-9ffe-dca9a00de929-kube-api-access-dqttc\") pod \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\" (UID: \"bf273e63-6dd2-49ea-9ffe-dca9a00de929\") " Nov 02 21:10:26.225818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.224961 5800 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-home\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.225818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.225089 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-home" pod "c0435af2-55cd-4cc3-b129-22a290db98ed" (UID: "c0435af2-55cd-4cc3-b129-22a290db98ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.225818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.225336 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-model-cache" pod "bf273e63-6dd2-49ea-9ffe-dca9a00de929" (UID: "bf273e63-6dd2-49ea-9ffe-dca9a00de929"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.225818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.225543 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-model-cache" pod "c0435af2-55cd-4cc3-b129-22a290db98ed" (UID: "c0435af2-55cd-4cc3-b129-22a290db98ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.225818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.225491 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tmp-dir" pod "bf273e63-6dd2-49ea-9ffe-dca9a00de929" (UID: "bf273e63-6dd2-49ea-9ffe-dca9a00de929"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.226381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.226360 5800 scope.go:122] "RemoveContainer" containerID="2bca9f0668540e5ada85a82313071845b2fdeb71d48e55dedcaef119a166fca4" Nov 02 21:10:26.226745 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.226691 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-dshm" pod "bf273e63-6dd2-49ea-9ffe-dca9a00de929" (UID: "bf273e63-6dd2-49ea-9ffe-dca9a00de929"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.226858 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.226752 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-tmp-dir" pod "c0435af2-55cd-4cc3-b129-22a290db98ed" (UID: "c0435af2-55cd-4cc3-b129-22a290db98ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.233438 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.229870 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf273e63-6dd2-49ea-9ffe-dca9a00de929-kube-api-access-dqttc" pod "bf273e63-6dd2-49ea-9ffe-dca9a00de929" (UID: "bf273e63-6dd2-49ea-9ffe-dca9a00de929"). InnerVolumeSpecName "kube-api-access-dqttc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:26.234428 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.234272 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-dshm" pod "c0435af2-55cd-4cc3-b129-22a290db98ed" (UID: "c0435af2-55cd-4cc3-b129-22a290db98ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:26.234428 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.234365 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0435af2-55cd-4cc3-b129-22a290db98ed-tls-certs" pod "c0435af2-55cd-4cc3-b129-22a290db98ed" (UID: "c0435af2-55cd-4cc3-b129-22a290db98ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:26.234747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.234438 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tls-certs" pod "bf273e63-6dd2-49ea-9ffe-dca9a00de929" (UID: "bf273e63-6dd2-49ea-9ffe-dca9a00de929"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:26.235781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.235678 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0435af2-55cd-4cc3-b129-22a290db98ed-kube-api-access-rpq44" pod "c0435af2-55cd-4cc3-b129-22a290db98ed" (UID: "c0435af2-55cd-4cc3-b129-22a290db98ed"). InnerVolumeSpecName "kube-api-access-rpq44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:26.331218 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331179 5800 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-dshm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331218 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331219 5800 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-model-cache\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331238 5800 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-model-cache\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331253 5800 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tls-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331267 5800 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-home\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331302 5800 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-dshm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331316 5800 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0435af2-55cd-4cc3-b129-22a290db98ed-tmp-dir\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331328 5800 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0435af2-55cd-4cc3-b129-22a290db98ed-tls-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331340 5800 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf273e63-6dd2-49ea-9ffe-dca9a00de929-tmp-dir\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331352 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqttc\" (UniqueName: \"kubernetes.io/projected/bf273e63-6dd2-49ea-9ffe-dca9a00de929-kube-api-access-dqttc\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.331495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.331366 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpq44\" (UniqueName: \"kubernetes.io/projected/c0435af2-55cd-4cc3-b129-22a290db98ed-kube-api-access-rpq44\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:26.450593 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.450554 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq"] Nov 02 21:10:26.452986 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:26.452953 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536a6fca_b3a8_4fc7_b212_8606774bfd24.slice/crio-f62a97caa1dbffe2340cc1505ceeda2f5025d8dc029126d9a85f5e589eec99b2 WatchSource:0}: Error finding container f62a97caa1dbffe2340cc1505ceeda2f5025d8dc029126d9a85f5e589eec99b2: Status 404 returned error can't find the container with id f62a97caa1dbffe2340cc1505ceeda2f5025d8dc029126d9a85f5e589eec99b2 Nov 02 21:10:26.494860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.494826 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" event={"ID":"bf273e63-6dd2-49ea-9ffe-dca9a00de929","Type":"ContainerDied","Data":"399abbd08706a5245015ada21011cd7b0b212f5b7b7fe46c6aa9d94d15ad962b"} Nov 02 21:10:26.494984 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.494870 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" event={"ID":"bf273e63-6dd2-49ea-9ffe-dca9a00de929","Type":"ContainerDied","Data":"ed96648eef74e8abdab5eb4d313dce812101d6e8b6c3354f8f0b052f45de033d"} Nov 02 21:10:26.496330 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.496276 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" event={"ID":"536a6fca-b3a8-4fc7-b212-8606774bfd24","Type":"ContainerStarted","Data":"f62a97caa1dbffe2340cc1505ceeda2f5025d8dc029126d9a85f5e589eec99b2"} Nov 02 21:10:26.496330 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.496310 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5" Nov 02 21:10:26.496529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.496335 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" event={"ID":"c0435af2-55cd-4cc3-b129-22a290db98ed","Type":"ContainerDied","Data":"2bca9f0668540e5ada85a82313071845b2fdeb71d48e55dedcaef119a166fca4"} Nov 02 21:10:26.496529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.496352 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" event={"ID":"c0435af2-55cd-4cc3-b129-22a290db98ed","Type":"ContainerDied","Data":"44689b0c4f12507660b5394b9d5fd817ebba6cfb0c8eec900012e808a9d5ec9c"} Nov 02 21:10:26.496529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.496315 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj" Nov 02 21:10:26.816356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.816317 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5"] Nov 02 21:10:26.821135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.821109 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-426941/distinct-w2-426941-kserve-78d6f46875-qzdm5"] Nov 02 21:10:26.922790 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.922708 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj"] Nov 02 21:10:26.925839 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:26.925807 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-426941/premium-w2-426941-kserve-6b7dbb77cb-vhdgj"] Nov 02 21:10:27.504807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:27.504777 5800 generic.go:356] "Generic (PLEG): container finished" podID="536a6fca-b3a8-4fc7-b212-8606774bfd24" containerID="9dbbd998623f424d8b298987a28d9f687b064fb29c4290b171609d8ab8c3e58f" exitCode=0 Nov 02 21:10:27.504988 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:27.504872 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" event={"ID":"536a6fca-b3a8-4fc7-b212-8606774bfd24","Type":"ContainerDied","Data":"9dbbd998623f424d8b298987a28d9f687b064fb29c4290b171609d8ab8c3e58f"} Nov 02 21:10:27.622471 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:27.622436 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" path="/var/lib/kubelet/pods/bf273e63-6dd2-49ea-9ffe-dca9a00de929/volumes" Nov 02 21:10:27.622908 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:27.622894 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" path="/var/lib/kubelet/pods/c0435af2-55cd-4cc3-b129-22a290db98ed/volumes" Nov 02 21:10:28.674589 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.674561 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:28.763640 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.763545 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536a6fca-b3a8-4fc7-b212-8606774bfd24-kube-api-access-rtj8r\" (UniqueName: \"kubernetes.io/projected/536a6fca-b3a8-4fc7-b212-8606774bfd24-kube-api-access-rtj8r\") pod \"536a6fca-b3a8-4fc7-b212-8606774bfd24\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " Nov 02 21:10:28.763640 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.763614 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-xks-ca\") pod \"536a6fca-b3a8-4fc7-b212-8606774bfd24\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " Nov 02 21:10:28.763891 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.763801 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-openshift-ca\") pod \"536a6fca-b3a8-4fc7-b212-8606774bfd24\" (UID: \"536a6fca-b3a8-4fc7-b212-8606774bfd24\") " Nov 02 21:10:28.764178 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.764147 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-openshift-ca" pod "536a6fca-b3a8-4fc7-b212-8606774bfd24" (UID: "536a6fca-b3a8-4fc7-b212-8606774bfd24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:28.764350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.764306 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:28.765799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.765759 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-xks-ca" pod "536a6fca-b3a8-4fc7-b212-8606774bfd24" (UID: "536a6fca-b3a8-4fc7-b212-8606774bfd24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:28.765955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.765931 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536a6fca-b3a8-4fc7-b212-8606774bfd24-kube-api-access-rtj8r" pod "536a6fca-b3a8-4fc7-b212-8606774bfd24" (UID: "536a6fca-b3a8-4fc7-b212-8606774bfd24"). InnerVolumeSpecName "kube-api-access-rtj8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:28.865960 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.865885 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtj8r\" (UniqueName: \"kubernetes.io/projected/536a6fca-b3a8-4fc7-b212-8606774bfd24-kube-api-access-rtj8r\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:28.865960 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:28.865916 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536a6fca-b3a8-4fc7-b212-8606774bfd24-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:29.520642 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:29.520612 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" Nov 02 21:10:29.520828 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:29.520612 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq" event={"ID":"536a6fca-b3a8-4fc7-b212-8606774bfd24","Type":"ContainerDied","Data":"f62a97caa1dbffe2340cc1505ceeda2f5025d8dc029126d9a85f5e589eec99b2"} Nov 02 21:10:29.520828 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:29.520737 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f62a97caa1dbffe2340cc1505ceeda2f5025d8dc029126d9a85f5e589eec99b2" Nov 02 21:10:31.915590 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.915556 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj"] Nov 02 21:10:31.916121 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916107 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" Nov 02 21:10:31.916174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916122 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" Nov 02 21:10:31.916174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916154 5800 cpu_manager.go:414] "containerMap: removing container" podUID="536a6fca-b3a8-4fc7-b212-8606774bfd24" containerName="revoke-keys" Nov 02 21:10:31.916174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916160 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536a6fca-b3a8-4fc7-b212-8606774bfd24" containerName="revoke-keys" Nov 02 21:10:31.916274 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916178 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" Nov 02 21:10:31.916274 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916184 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" Nov 02 21:10:31.916274 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916264 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="536a6fca-b3a8-4fc7-b212-8606774bfd24" containerName="revoke-keys" Nov 02 21:10:31.916391 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916290 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0435af2-55cd-4cc3-b129-22a290db98ed" containerName="main" Nov 02 21:10:31.916391 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.916299 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf273e63-6dd2-49ea-9ffe-dca9a00de929" containerName="main" Nov 02 21:10:31.919529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.919511 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:31.968562 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.968521 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:10:31.971855 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:31.971826 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj"] Nov 02 21:10:32.001852 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.001443 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shms\" (UniqueName: \"kubernetes.io/projected/c8245b1a-ba35-4453-882c-562f7ee70e17-kube-api-access-6shms\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.001852 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.001527 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.001852 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.001587 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.014272 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.014234 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:10:32.014941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.014922 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:10:32.015213 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.015199 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:10:32.102383 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.102346 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shms\" (UniqueName: \"kubernetes.io/projected/c8245b1a-ba35-4453-882c-562f7ee70e17-kube-api-access-6shms\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.102620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.102420 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.102620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.102484 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.103134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.103091 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.104946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.104921 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.155703 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.155668 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shms\" (UniqueName: \"kubernetes.io/projected/c8245b1a-ba35-4453-882c-562f7ee70e17-kube-api-access-6shms\") pod \"maas-api-revoke-sub-7ec908d9-8wgmj\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.234232 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.234150 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:32.640277 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.640247 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj"] Nov 02 21:10:32.642157 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:32.642121 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8245b1a_ba35_4453_882c_562f7ee70e17.slice/crio-c6ec123240cd1c55b2809cd00b9b12c9a9e81db8dfeae24f858a7485325124fc WatchSource:0}: Error finding container c6ec123240cd1c55b2809cd00b9b12c9a9e81db8dfeae24f858a7485325124fc: Status 404 returned error can't find the container with id c6ec123240cd1c55b2809cd00b9b12c9a9e81db8dfeae24f858a7485325124fc Nov 02 21:10:32.691434 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:32.691399 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" podUID="83840101-e53a-436f-ae00-bcdd34063062" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:10:33.574628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:33.574583 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" event={"ID":"c8245b1a-ba35-4453-882c-562f7ee70e17","Type":"ContainerStarted","Data":"76f70424e73638034d0818e6e9cdfac9cb6ad8c65b9707e1ed3d43af88ce82a1"} Nov 02 21:10:33.575046 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:33.574642 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" event={"ID":"c8245b1a-ba35-4453-882c-562f7ee70e17","Type":"ContainerStarted","Data":"c6ec123240cd1c55b2809cd00b9b12c9a9e81db8dfeae24f858a7485325124fc"} Nov 02 21:10:33.681919 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:33.681865 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" podStartSLOduration=2.681846129 podStartE2EDuration="2.681846129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:33.680444884 +0000 UTC m=+4932.197217469" watchObservedRunningTime="2026-11-02 21:10:33.681846129 +0000 UTC m=+4932.198618715" Nov 02 21:10:34.584800 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:34.584762 5800 generic.go:356] "Generic (PLEG): container finished" podID="c8245b1a-ba35-4453-882c-562f7ee70e17" containerID="76f70424e73638034d0818e6e9cdfac9cb6ad8c65b9707e1ed3d43af88ce82a1" exitCode=0 Nov 02 21:10:34.585230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:34.584858 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" event={"ID":"c8245b1a-ba35-4453-882c-562f7ee70e17","Type":"ContainerDied","Data":"76f70424e73638034d0818e6e9cdfac9cb6ad8c65b9707e1ed3d43af88ce82a1"} Nov 02 21:10:35.040856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.040822 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx"] Nov 02 21:10:35.041042 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.041022 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" containerID="cri-o://5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce" gracePeriod=30 Nov 02 21:10:35.086933 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.086889 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp"] Nov 02 21:10:35.087117 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.087095 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" containerID="cri-o://2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e" gracePeriod=30 Nov 02 21:10:35.090259 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.090230 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7"] Nov 02 21:10:35.090511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.090481 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" containerID="cri-o://1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386" gracePeriod=30 Nov 02 21:10:35.094137 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.094114 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42"] Nov 02 21:10:35.094388 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.094366 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" containerID="cri-o://2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e" gracePeriod=30 Nov 02 21:10:35.742049 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.742013 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:35.878679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.878594 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8245b1a-ba35-4453-882c-562f7ee70e17-kube-api-access-6shms\" (UniqueName: \"kubernetes.io/projected/c8245b1a-ba35-4453-882c-562f7ee70e17-kube-api-access-6shms\") pod \"c8245b1a-ba35-4453-882c-562f7ee70e17\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " Nov 02 21:10:35.878830 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.878742 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-xks-ca\") pod \"c8245b1a-ba35-4453-882c-562f7ee70e17\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " Nov 02 21:10:35.878830 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.878781 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-openshift-ca\") pod \"c8245b1a-ba35-4453-882c-562f7ee70e17\" (UID: \"c8245b1a-ba35-4453-882c-562f7ee70e17\") " Nov 02 21:10:35.879194 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.879170 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-openshift-ca" pod "c8245b1a-ba35-4453-882c-562f7ee70e17" (UID: "c8245b1a-ba35-4453-882c-562f7ee70e17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:35.880956 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.880927 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8245b1a-ba35-4453-882c-562f7ee70e17-kube-api-access-6shms" pod "c8245b1a-ba35-4453-882c-562f7ee70e17" (UID: "c8245b1a-ba35-4453-882c-562f7ee70e17"). InnerVolumeSpecName "kube-api-access-6shms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:35.881247 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.881219 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-xks-ca" pod "c8245b1a-ba35-4453-882c-562f7ee70e17" (UID: "c8245b1a-ba35-4453-882c-562f7ee70e17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:35.980610 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.980564 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shms\" (UniqueName: \"kubernetes.io/projected/c8245b1a-ba35-4453-882c-562f7ee70e17-kube-api-access-6shms\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:35.980610 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.980605 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:35.980610 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:35.980616 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8245b1a-ba35-4453-882c-562f7ee70e17-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:36.048415 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.048377 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr"] Nov 02 21:10:36.049079 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.049057 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c8245b1a-ba35-4453-882c-562f7ee70e17" containerName="revoke-keys" Nov 02 21:10:36.049193 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.049080 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8245b1a-ba35-4453-882c-562f7ee70e17" containerName="revoke-keys" Nov 02 21:10:36.049276 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.049217 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8245b1a-ba35-4453-882c-562f7ee70e17" containerName="revoke-keys" Nov 02 21:10:36.052860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.052838 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.106237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.106199 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr"] Nov 02 21:10:36.182225 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.182127 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm69m\" (UniqueName: \"kubernetes.io/projected/67819d19-c365-43e4-bc81-1350237bd39c-kube-api-access-rm69m\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.182447 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.182260 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.182447 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.182345 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.283357 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.283323 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm69m\" (UniqueName: \"kubernetes.io/projected/67819d19-c365-43e4-bc81-1350237bd39c-kube-api-access-rm69m\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.283517 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.283400 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.283517 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.283444 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.284181 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.284155 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.285834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.285813 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.363330 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.363222 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm69m\" (UniqueName: \"kubernetes.io/projected/67819d19-c365-43e4-bc81-1350237bd39c-kube-api-access-rm69m\") pod \"maas-api-revoke-sub-2189fd46-rvpmr\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.368867 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.368827 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:36.600112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.600084 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" Nov 02 21:10:36.600309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.600083 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj" event={"ID":"c8245b1a-ba35-4453-882c-562f7ee70e17","Type":"ContainerDied","Data":"c6ec123240cd1c55b2809cd00b9b12c9a9e81db8dfeae24f858a7485325124fc"} Nov 02 21:10:36.600309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.600204 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ec123240cd1c55b2809cd00b9b12c9a9e81db8dfeae24f858a7485325124fc" Nov 02 21:10:36.644401 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.644368 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7"] Nov 02 21:10:36.649723 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.649702 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.700757 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.700720 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7"] Nov 02 21:10:36.742007 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:36.741976 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67819d19_c365_43e4_bc81_1350237bd39c.slice/crio-e6c054bfbaae75cdcf5b91d449cec4dec3f6d8c8ae60d55448a88aa3fba27611 WatchSource:0}: Error finding container e6c054bfbaae75cdcf5b91d449cec4dec3f6d8c8ae60d55448a88aa3fba27611: Status 404 returned error can't find the container with id e6c054bfbaae75cdcf5b91d449cec4dec3f6d8c8ae60d55448a88aa3fba27611 Nov 02 21:10:36.743448 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.743418 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr"] Nov 02 21:10:36.789664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.789628 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.789832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.789793 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.789986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.789970 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2r2m\" (UniqueName: \"kubernetes.io/projected/47d7595b-13ba-4348-b5ee-323a5e1e8c44-kube-api-access-c2r2m\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.891438 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.891347 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.891627 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.891504 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2r2m\" (UniqueName: \"kubernetes.io/projected/47d7595b-13ba-4348-b5ee-323a5e1e8c44-kube-api-access-c2r2m\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.891627 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.891543 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.892208 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.892187 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.893835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.893813 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.944521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.944491 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2r2m\" (UniqueName: \"kubernetes.io/projected/47d7595b-13ba-4348-b5ee-323a5e1e8c44-kube-api-access-c2r2m\") pod \"maas-api-revoke-sub-336b1968-s2gh7\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:36.964316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:36.964250 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:37.107477 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.107437 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:10:37.107658 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.107504 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:10:37.385195 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.385164 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7"] Nov 02 21:10:37.386475 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:37.386447 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d7595b_13ba_4348_b5ee_323a5e1e8c44.slice/crio-33d8e5c8c073a9ff8b6b196e126ed71f971c356753ccfc9c06c10a5fba290c15 WatchSource:0}: Error finding container 33d8e5c8c073a9ff8b6b196e126ed71f971c356753ccfc9c06c10a5fba290c15: Status 404 returned error can't find the container with id 33d8e5c8c073a9ff8b6b196e126ed71f971c356753ccfc9c06c10a5fba290c15 Nov 02 21:10:37.530183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.530114 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:10:37.530413 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.530169 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:10:37.609514 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.609464 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" event={"ID":"67819d19-c365-43e4-bc81-1350237bd39c","Type":"ContainerStarted","Data":"f65b356810304983d8395034fde1da2f9378a11dfa269fc75d756709ed579650"} Nov 02 21:10:37.609727 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.609544 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" event={"ID":"67819d19-c365-43e4-bc81-1350237bd39c","Type":"ContainerStarted","Data":"e6c054bfbaae75cdcf5b91d449cec4dec3f6d8c8ae60d55448a88aa3fba27611"} Nov 02 21:10:37.611419 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.611382 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" event={"ID":"47d7595b-13ba-4348-b5ee-323a5e1e8c44","Type":"ContainerStarted","Data":"33d8e5c8c073a9ff8b6b196e126ed71f971c356753ccfc9c06c10a5fba290c15"} Nov 02 21:10:37.717929 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:37.717880 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" podStartSLOduration=1.717865825 podStartE2EDuration="1.717865825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:10:37.715391183 +0000 UTC m=+4936.232163768" watchObservedRunningTime="2026-11-02 21:10:37.717865825 +0000 UTC m=+4936.234638447" Nov 02 21:10:38.546647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.546602 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:10:38.547122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.546667 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:10:38.622509 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.622471 5800 generic.go:356] "Generic (PLEG): container finished" podID="47d7595b-13ba-4348-b5ee-323a5e1e8c44" containerID="2cd9169a029ca60b6e4555dfdbe5b0d8caf9e9a1f4b5ee4c5a39a3cba3a165a9" exitCode=0 Nov 02 21:10:38.622682 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.622554 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" event={"ID":"47d7595b-13ba-4348-b5ee-323a5e1e8c44","Type":"ContainerDied","Data":"2cd9169a029ca60b6e4555dfdbe5b0d8caf9e9a1f4b5ee4c5a39a3cba3a165a9"} Nov 02 21:10:38.624386 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.624360 5800 generic.go:356] "Generic (PLEG): container finished" podID="67819d19-c365-43e4-bc81-1350237bd39c" containerID="f65b356810304983d8395034fde1da2f9378a11dfa269fc75d756709ed579650" exitCode=0 Nov 02 21:10:38.624523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.624415 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" event={"ID":"67819d19-c365-43e4-bc81-1350237bd39c","Type":"ContainerDied","Data":"f65b356810304983d8395034fde1da2f9378a11dfa269fc75d756709ed579650"} Nov 02 21:10:38.728208 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.728159 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-6bv42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:10:38.728543 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:38.728219 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:10:39.808626 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.808386 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:39.847822 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.847790 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:39.926312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.926210 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-xks-ca\") pod \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " Nov 02 21:10:39.926312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.926302 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-openshift-ca\") pod \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " Nov 02 21:10:39.926565 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.926396 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d7595b-13ba-4348-b5ee-323a5e1e8c44-kube-api-access-c2r2m\" (UniqueName: \"kubernetes.io/projected/47d7595b-13ba-4348-b5ee-323a5e1e8c44-kube-api-access-c2r2m\") pod \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\" (UID: \"47d7595b-13ba-4348-b5ee-323a5e1e8c44\") " Nov 02 21:10:39.926716 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.926692 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-openshift-ca" pod "47d7595b-13ba-4348-b5ee-323a5e1e8c44" (UID: "47d7595b-13ba-4348-b5ee-323a5e1e8c44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:39.926904 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.926886 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:39.928840 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.928805 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d7595b-13ba-4348-b5ee-323a5e1e8c44-kube-api-access-c2r2m" pod "47d7595b-13ba-4348-b5ee-323a5e1e8c44" (UID: "47d7595b-13ba-4348-b5ee-323a5e1e8c44"). InnerVolumeSpecName "kube-api-access-c2r2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:39.928970 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:39.928856 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-xks-ca" pod "47d7595b-13ba-4348-b5ee-323a5e1e8c44" (UID: "47d7595b-13ba-4348-b5ee-323a5e1e8c44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:40.027832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.027793 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-openshift-ca\") pod \"67819d19-c365-43e4-bc81-1350237bd39c\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " Nov 02 21:10:40.028001 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.027939 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-xks-ca\") pod \"67819d19-c365-43e4-bc81-1350237bd39c\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " Nov 02 21:10:40.028081 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.028065 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67819d19-c365-43e4-bc81-1350237bd39c-kube-api-access-rm69m\" (UniqueName: \"kubernetes.io/projected/67819d19-c365-43e4-bc81-1350237bd39c-kube-api-access-rm69m\") pod \"67819d19-c365-43e4-bc81-1350237bd39c\" (UID: \"67819d19-c365-43e4-bc81-1350237bd39c\") " Nov 02 21:10:40.028200 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.028173 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-openshift-ca" pod "67819d19-c365-43e4-bc81-1350237bd39c" (UID: "67819d19-c365-43e4-bc81-1350237bd39c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:40.028571 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.028544 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47d7595b-13ba-4348-b5ee-323a5e1e8c44-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.028571 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.028570 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2r2m\" (UniqueName: \"kubernetes.io/projected/47d7595b-13ba-4348-b5ee-323a5e1e8c44-kube-api-access-c2r2m\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.028767 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.028584 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.030354 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.030320 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-xks-ca" pod "67819d19-c365-43e4-bc81-1350237bd39c" (UID: "67819d19-c365-43e4-bc81-1350237bd39c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:40.030478 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.030428 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67819d19-c365-43e4-bc81-1350237bd39c-kube-api-access-rm69m" pod "67819d19-c365-43e4-bc81-1350237bd39c" (UID: "67819d19-c365-43e4-bc81-1350237bd39c"). InnerVolumeSpecName "kube-api-access-rm69m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.129906 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.129866 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67819d19-c365-43e4-bc81-1350237bd39c-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.129906 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.129900 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm69m\" (UniqueName: \"kubernetes.io/projected/67819d19-c365-43e4-bc81-1350237bd39c-kube-api-access-rm69m\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.371339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.371267 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 21:10:40.411908 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.411876 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 21:10:40.424439 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.424409 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 21:10:40.465772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.465747 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 21:10:40.534837 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.534804 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378a9e33-b189-43ec-a595-195a873d45e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378a9e33-b189-43ec-a595-195a873d45e1-istiod-ca-cert\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.534837 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.534841 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-kube-api-access-hd6lh\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-kube-api-access-hd6lh\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.535099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.534865 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-envoy\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.535099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.534889 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-credential-socket\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.535099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.534923 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-envoy\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.535099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.534959 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-data\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.535099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.534996 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-credential-socket\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.535099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535028 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-envoy\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.535099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535053 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-certs\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535134 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/378a9e33-b189-43ec-a595-195a873d45e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378a9e33-b189-43ec-a595-195a873d45e1-istio-podinfo\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535160 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-token\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-token\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535197 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-data\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535257 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378a9e33-b189-43ec-a595-195a873d45e1-istiod-ca-cert" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535275 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-certs\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535303 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-data" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535322 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-certs\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535369 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-data\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535405 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-kube-api-access-lf66s\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-kube-api-access-lf66s\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535442 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istiod-ca-cert\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535461 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-credential-socket" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535489 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-token\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-token\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.535579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535554 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-credential-socket\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535613 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-kube-api-access-vg6ls\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-kube-api-access-vg6ls\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535674 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-podinfo\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.535702 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-socket\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.536074 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-credential-socket" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.536212 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-certs" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.536361 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istiod-ca-cert" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.536507 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-certs" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.536721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.536684 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-data" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.537181 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537090 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-data" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.537371 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537334 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-certs" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.537608 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537569 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-socket" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.537678 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537663 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-podinfo\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.537741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537696 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-socket\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.537741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537723 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-socket\") pod \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\" (UID: \"0704c06f-bf67-4bf7-9a4b-2300e00f14e7\") " Nov 02 21:10:40.537848 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537755 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istiod-ca-cert\") pod \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\" (UID: \"4c7a1b5f-0d72-480b-b514-f8cee2f619c8\") " Nov 02 21:10:40.537848 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.537804 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-istio-token\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-istio-token\") pod \"378a9e33-b189-43ec-a595-195a873d45e1\" (UID: \"378a9e33-b189-43ec-a595-195a873d45e1\") " Nov 02 21:10:40.538210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538185 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538206 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378a9e33-b189-43ec-a595-195a873d45e1-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538220 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538233 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538244 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538257 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538269 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538300 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538313 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538326 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538339 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.538462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538342 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-envoy" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.539026 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538796 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-credential-socket" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.539026 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.538862 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-token" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.539153 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.539120 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-socket" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.539153 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.539131 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-socket" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.539325 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.539228 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-kube-api-access-hd6lh" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "kube-api-access-hd6lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.539422 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.539391 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istiod-ca-cert" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:40.540248 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.540197 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-envoy" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.540435 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.540398 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-kube-api-access-lf66s" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "kube-api-access-lf66s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.540522 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.540506 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-podinfo" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:10:40.540770 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.540743 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-istio-token" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.541115 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.541090 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-token" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.541219 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.541203 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-envoy" pod "4c7a1b5f-0d72-480b-b514-f8cee2f619c8" (UID: "4c7a1b5f-0d72-480b-b514-f8cee2f619c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.541620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.541600 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/378a9e33-b189-43ec-a595-195a873d45e1-istio-podinfo" pod "378a9e33-b189-43ec-a595-195a873d45e1" (UID: "378a9e33-b189-43ec-a595-195a873d45e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:10:40.541927 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.541913 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-kube-api-access-vg6ls" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "kube-api-access-vg6ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.542127 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.542108 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-podinfo" pod "0704c06f-bf67-4bf7-9a4b-2300e00f14e7" (UID: "0704c06f-bf67-4bf7-9a4b-2300e00f14e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:10:40.639081 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639046 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-data\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.639346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639092 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775c780e-81ac-4b0e-a286-66724a1e86f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c780e-81ac-4b0e-a286-66724a1e86f0-istiod-ca-cert\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.639346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639134 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-certs\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.639346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639188 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-credential-socket\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.639724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639420 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-data" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.639724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639495 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-envoy\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.639724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639509 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-credential-socket" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.639724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639599 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-token\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-token\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.639724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639700 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-kube-api-access-gd7wf\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-kube-api-access-gd7wf\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.640055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639737 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-socket\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.640055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.639778 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-podinfo\") pod \"775c780e-81ac-4b0e-a286-66724a1e86f0\" (UID: \"775c780e-81ac-4b0e-a286-66724a1e86f0\") " Nov 02 21:10:40.640165 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640140 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-socket" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.640356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640333 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd6lh\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-kube-api-access-hd6lh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640363 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640380 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640396 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640410 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640424 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640438 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378a9e33-b189-43ec-a595-195a873d45e1-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640452 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640465 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf66s\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-kube-api-access-lf66s\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640479 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640493 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640508 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6ls\" (UniqueName: \"kubernetes.io/projected/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-kube-api-access-vg6ls\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640521 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640536 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640550 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640563 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378a9e33-b189-43ec-a595-195a873d45e1-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640577 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0704c06f-bf67-4bf7-9a4b-2300e00f14e7-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640591 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7a1b5f-0d72-480b-b514-f8cee2f619c8-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.640649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640605 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378a9e33-b189-43ec-a595-195a873d45e1-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.641146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640689 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-certs" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.641146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.640855 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775c780e-81ac-4b0e-a286-66724a1e86f0-istiod-ca-cert" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:40.642322 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.642295 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-podinfo" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:10:40.642530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.642503 5800 generic.go:356] "Generic (PLEG): container finished" podID="378a9e33-b189-43ec-a595-195a873d45e1" containerID="2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e" exitCode=0 Nov 02 21:10:40.642530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.642517 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-envoy" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:10:40.642698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.642558 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" Nov 02 21:10:40.642698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.642584 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" event={"ID":"378a9e33-b189-43ec-a595-195a873d45e1","Type":"ContainerDied","Data":"2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e"} Nov 02 21:10:40.642698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.642641 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp" event={"ID":"378a9e33-b189-43ec-a595-195a873d45e1","Type":"ContainerDied","Data":"59cfd3ddde435c4321d4a0dc8c7e6bb8944d85da3ce03a33acc9e8ff0539f9f0"} Nov 02 21:10:40.642698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.642662 5800 scope.go:122] "RemoveContainer" containerID="2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e" Nov 02 21:10:40.643045 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.643015 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-kube-api-access-gd7wf" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "kube-api-access-gd7wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.643157 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.643116 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-token" pod "775c780e-81ac-4b0e-a286-66724a1e86f0" (UID: "775c780e-81ac-4b0e-a286-66724a1e86f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:40.644476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.644449 5800 generic.go:356] "Generic (PLEG): container finished" podID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerID="1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386" exitCode=0 Nov 02 21:10:40.644587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.644500 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" Nov 02 21:10:40.644587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.644497 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" event={"ID":"775c780e-81ac-4b0e-a286-66724a1e86f0","Type":"ContainerDied","Data":"1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386"} Nov 02 21:10:40.644587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.644553 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7" event={"ID":"775c780e-81ac-4b0e-a286-66724a1e86f0","Type":"ContainerDied","Data":"6c9684dbe04232a51388afa2ef0d37aca3728b7db1ab157a963f2afbf0eeeb67"} Nov 02 21:10:40.646256 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.646236 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" Nov 02 21:10:40.646384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.646300 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr" event={"ID":"67819d19-c365-43e4-bc81-1350237bd39c","Type":"ContainerDied","Data":"e6c054bfbaae75cdcf5b91d449cec4dec3f6d8c8ae60d55448a88aa3fba27611"} Nov 02 21:10:40.646384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.646350 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c054bfbaae75cdcf5b91d449cec4dec3f6d8c8ae60d55448a88aa3fba27611" Nov 02 21:10:40.659550 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.655728 5800 generic.go:356] "Generic (PLEG): container finished" podID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerID="5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce" exitCode=0 Nov 02 21:10:40.659550 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.655861 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" Nov 02 21:10:40.659550 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.656206 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" event={"ID":"0704c06f-bf67-4bf7-9a4b-2300e00f14e7","Type":"ContainerDied","Data":"5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce"} Nov 02 21:10:40.659550 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.656268 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx" event={"ID":"0704c06f-bf67-4bf7-9a4b-2300e00f14e7","Type":"ContainerDied","Data":"8038fa68869002f0d9e410d489b5b20ee4590f9a496e4cf72b588d13df42ebcb"} Nov 02 21:10:40.660861 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.660825 5800 scope.go:122] "RemoveContainer" containerID="2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e" Nov 02 21:10:40.661716 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:40.661688 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e\": container with ID starting with 2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e not found: ID does not exist" containerID="2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e" Nov 02 21:10:40.661829 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.661726 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e"} err="failed to get container status \"2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e\": rpc error: code = NotFound desc = could not find container \"2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e\": container with ID starting with 2edb6543cc5e68d6ea03196728ca457047ef9901eed0fd6107b60b6617e2162e not found: ID does not exist" Nov 02 21:10:40.661829 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.661752 5800 scope.go:122] "RemoveContainer" containerID="1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386" Nov 02 21:10:40.669331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.663378 5800 generic.go:356] "Generic (PLEG): container finished" podID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerID="2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e" exitCode=0 Nov 02 21:10:40.669331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.663460 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" event={"ID":"4c7a1b5f-0d72-480b-b514-f8cee2f619c8","Type":"ContainerDied","Data":"2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e"} Nov 02 21:10:40.669331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.663507 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" event={"ID":"4c7a1b5f-0d72-480b-b514-f8cee2f619c8","Type":"ContainerDied","Data":"d4410f74a6b9e43ec64c0d108ac35135796d8e9dfeb9f541e0e1f759f507ad46"} Nov 02 21:10:40.669331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.663578 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42" Nov 02 21:10:40.669331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.665637 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" event={"ID":"47d7595b-13ba-4348-b5ee-323a5e1e8c44","Type":"ContainerDied","Data":"33d8e5c8c073a9ff8b6b196e126ed71f971c356753ccfc9c06c10a5fba290c15"} Nov 02 21:10:40.669331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.665670 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33d8e5c8c073a9ff8b6b196e126ed71f971c356753ccfc9c06c10a5fba290c15" Nov 02 21:10:40.669331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.665717 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7" Nov 02 21:10:40.678561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.678539 5800 scope.go:122] "RemoveContainer" containerID="1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386" Nov 02 21:10:40.678849 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:40.678821 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386\": container with ID starting with 1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386 not found: ID does not exist" containerID="1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386" Nov 02 21:10:40.678920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.678857 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386"} err="failed to get container status \"1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386\": rpc error: code = NotFound desc = could not find container \"1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386\": container with ID starting with 1343fb94a5af81436d9f791f9790cc6a3f4508dff5704bd4d9099c8c7b130386 not found: ID does not exist" Nov 02 21:10:40.678920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.678873 5800 scope.go:122] "RemoveContainer" containerID="5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce" Nov 02 21:10:40.688166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.688146 5800 scope.go:122] "RemoveContainer" containerID="5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce" Nov 02 21:10:40.688479 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:40.688457 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce\": container with ID starting with 5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce not found: ID does not exist" containerID="5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce" Nov 02 21:10:40.688579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.688492 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce"} err="failed to get container status \"5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce\": rpc error: code = NotFound desc = could not find container \"5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce\": container with ID starting with 5452e8d81759014d2ee144631d061ca38236e003716af9ccea0b1e172d8445ce not found: ID does not exist" Nov 02 21:10:40.688579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.688517 5800 scope.go:122] "RemoveContainer" containerID="2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e" Nov 02 21:10:40.698172 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.698147 5800 scope.go:122] "RemoveContainer" containerID="2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e" Nov 02 21:10:40.698556 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:40.698531 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e\": container with ID starting with 2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e not found: ID does not exist" containerID="2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e" Nov 02 21:10:40.698663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.698564 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e"} err="failed to get container status \"2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e\": rpc error: code = NotFound desc = could not find container \"2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e\": container with ID starting with 2b182f55653adb5b78d258aaef656b4fba8cd9481ed286c60d897e9ff5297e7e not found: ID does not exist" Nov 02 21:10:40.742230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.741920 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/775c780e-81ac-4b0e-a286-66724a1e86f0-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.742230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.741947 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.742230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.741957 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.742230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.741966 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.742230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.741977 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd7wf\" (UniqueName: \"kubernetes.io/projected/775c780e-81ac-4b0e-a286-66724a1e86f0-kube-api-access-gd7wf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:40.742230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:40.741987 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/775c780e-81ac-4b0e-a286-66724a1e86f0-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:41.015416 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.015382 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp"] Nov 02 21:10:41.019483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.019458 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-qt5rp"] Nov 02 21:10:41.122519 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.122482 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7"] Nov 02 21:10:41.125358 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.125333 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-b4gn7"] Nov 02 21:10:41.228065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.228034 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx"] Nov 02 21:10:41.234096 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.234068 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-wfqpx"] Nov 02 21:10:41.335069 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.334988 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42"] Nov 02 21:10:41.341505 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.341471 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-6bv42"] Nov 02 21:10:41.623471 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.623385 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" path="/var/lib/kubelet/pods/0704c06f-bf67-4bf7-9a4b-2300e00f14e7/volumes" Nov 02 21:10:41.623989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.623967 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378a9e33-b189-43ec-a595-195a873d45e1" path="/var/lib/kubelet/pods/378a9e33-b189-43ec-a595-195a873d45e1/volumes" Nov 02 21:10:41.624505 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.624485 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" path="/var/lib/kubelet/pods/4c7a1b5f-0d72-480b-b514-f8cee2f619c8/volumes" Nov 02 21:10:41.625373 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:41.625350 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" path="/var/lib/kubelet/pods/775c780e-81ac-4b0e-a286-66724a1e86f0/volumes" Nov 02 21:10:42.692648 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.692612 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" podUID="83840101-e53a-436f-ae00-bcdd34063062" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:10:42.832413 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.832375 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc"] Nov 02 21:10:42.832948 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.832934 5800 cpu_manager.go:414] "containerMap: removing container" podUID="67819d19-c365-43e4-bc81-1350237bd39c" containerName="revoke-keys" Nov 02 21:10:42.832995 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.832951 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67819d19-c365-43e4-bc81-1350237bd39c" containerName="revoke-keys" Nov 02 21:10:42.832995 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.832974 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" Nov 02 21:10:42.832995 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.832981 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.832999 5800 cpu_manager.go:414] "containerMap: removing container" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833004 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833011 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833016 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833024 5800 cpu_manager.go:414] "containerMap: removing container" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833028 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833039 5800 cpu_manager.go:414] "containerMap: removing container" podUID="47d7595b-13ba-4348-b5ee-323a5e1e8c44" containerName="revoke-keys" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833044 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d7595b-13ba-4348-b5ee-323a5e1e8c44" containerName="revoke-keys" Nov 02 21:10:42.833116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833119 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d7595b-13ba-4348-b5ee-323a5e1e8c44" containerName="revoke-keys" Nov 02 21:10:42.833427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833130 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="67819d19-c365-43e4-bc81-1350237bd39c" containerName="revoke-keys" Nov 02 21:10:42.833427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833139 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7a1b5f-0d72-480b-b514-f8cee2f619c8" containerName="istio-proxy" Nov 02 21:10:42.833427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833147 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="378a9e33-b189-43ec-a595-195a873d45e1" containerName="istio-proxy" Nov 02 21:10:42.833427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833153 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0704c06f-bf67-4bf7-9a4b-2300e00f14e7" containerName="istio-proxy" Nov 02 21:10:42.833427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.833158 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c780e-81ac-4b0e-a286-66724a1e86f0" containerName="istio-proxy" Nov 02 21:10:42.837739 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.837722 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.868028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.867994 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.868211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.868143 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.868211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.868175 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tlh\" (UniqueName: \"kubernetes.io/projected/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-kube-api-access-p4tlh\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.890177 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.890146 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc"] Nov 02 21:10:42.932346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.932264 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:10:42.932346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.932277 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:10:42.932604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.932375 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:10:42.932684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.932627 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:10:42.969370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.969265 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.969370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.969348 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tlh\" (UniqueName: \"kubernetes.io/projected/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-kube-api-access-p4tlh\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.969564 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.969387 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.970038 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.970017 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:42.971897 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:42.971874 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:43.022667 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:43.022626 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tlh\" (UniqueName: \"kubernetes.io/projected/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-kube-api-access-p4tlh\") pod \"maas-api-revoke-sub-f62b26cc-qs4dc\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:43.153667 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:43.153628 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:43.522512 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:43.522479 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc"] Nov 02 21:10:43.524492 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:43.524465 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b83fd7_15a0_4f8d_b144_7ef56559e77f.slice/crio-b80da1f294d1fc8b2dddf4a733428132699463db0016034b1deaa72aecd9544c WatchSource:0}: Error finding container b80da1f294d1fc8b2dddf4a733428132699463db0016034b1deaa72aecd9544c: Status 404 returned error can't find the container with id b80da1f294d1fc8b2dddf4a733428132699463db0016034b1deaa72aecd9544c Nov 02 21:10:43.698495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:43.698459 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" event={"ID":"b0b83fd7-15a0-4f8d-b144-7ef56559e77f","Type":"ContainerStarted","Data":"b80da1f294d1fc8b2dddf4a733428132699463db0016034b1deaa72aecd9544c"} Nov 02 21:10:44.595724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.595664 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 21:10:44.709996 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.709956 5800 generic.go:356] "Generic (PLEG): container finished" podID="83840101-e53a-436f-ae00-bcdd34063062" containerID="ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4" exitCode=0 Nov 02 21:10:44.709996 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.710000 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" Nov 02 21:10:44.710625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.709996 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" event={"ID":"83840101-e53a-436f-ae00-bcdd34063062","Type":"ContainerDied","Data":"ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4"} Nov 02 21:10:44.710625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.710068 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9" event={"ID":"83840101-e53a-436f-ae00-bcdd34063062","Type":"ContainerDied","Data":"873043528cd33d5e30ce8cafd6dcf9e7ef9a06de9028f1bc5aed68d9642ab490"} Nov 02 21:10:44.710625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.710101 5800 scope.go:122] "RemoveContainer" containerID="ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4" Nov 02 21:10:44.712066 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.712028 5800 generic.go:356] "Generic (PLEG): container finished" podID="b0b83fd7-15a0-4f8d-b144-7ef56559e77f" containerID="4c81d04e11e93e515c1639e0458161230c6240982759ddec3232dbbc98362bb5" exitCode=0 Nov 02 21:10:44.712206 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.712150 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" event={"ID":"b0b83fd7-15a0-4f8d-b144-7ef56559e77f","Type":"ContainerDied","Data":"4c81d04e11e93e515c1639e0458161230c6240982759ddec3232dbbc98362bb5"} Nov 02 21:10:44.723668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.723633 5800 scope.go:122] "RemoveContainer" containerID="ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4" Nov 02 21:10:44.723944 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:44.723919 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4\": container with ID starting with ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4 not found: ID does not exist" containerID="ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4" Nov 02 21:10:44.724049 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.723960 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4"} err="failed to get container status \"ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4\": rpc error: code = NotFound desc = could not find container \"ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4\": container with ID starting with ce2ee855f7214763843c9980faa10d565a26c56da0fd259dcad4181beb3529b4 not found: ID does not exist" Nov 02 21:10:44.784712 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.784674 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83840101-e53a-436f-ae00-bcdd34063062-kube-api-access-pmr5n\" (UniqueName: \"kubernetes.io/projected/83840101-e53a-436f-ae00-bcdd34063062-kube-api-access-pmr5n\") pod \"83840101-e53a-436f-ae00-bcdd34063062\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " Nov 02 21:10:44.784919 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.784803 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83840101-e53a-436f-ae00-bcdd34063062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83840101-e53a-436f-ae00-bcdd34063062-plugins-config-volume\") pod \"83840101-e53a-436f-ae00-bcdd34063062\" (UID: \"83840101-e53a-436f-ae00-bcdd34063062\") " Nov 02 21:10:44.785172 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.785149 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83840101-e53a-436f-ae00-bcdd34063062-plugins-config-volume" pod "83840101-e53a-436f-ae00-bcdd34063062" (UID: "83840101-e53a-436f-ae00-bcdd34063062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:44.786921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.786899 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83840101-e53a-436f-ae00-bcdd34063062-kube-api-access-pmr5n" pod "83840101-e53a-436f-ae00-bcdd34063062" (UID: "83840101-e53a-436f-ae00-bcdd34063062"). InnerVolumeSpecName "kube-api-access-pmr5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:44.885763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.885678 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83840101-e53a-436f-ae00-bcdd34063062-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:44.885763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:44.885708 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmr5n\" (UniqueName: \"kubernetes.io/projected/83840101-e53a-436f-ae00-bcdd34063062-kube-api-access-pmr5n\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:45.125714 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.125674 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9"] Nov 02 21:10:45.129011 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.128981 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-426941-6947599fd7-cvkk9"] Nov 02 21:10:45.622801 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.622769 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83840101-e53a-436f-ae00-bcdd34063062" path="/var/lib/kubelet/pods/83840101-e53a-436f-ae00-bcdd34063062/volumes" Nov 02 21:10:45.878308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.878215 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:45.893723 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.893692 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-xks-ca\") pod \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " Nov 02 21:10:45.893856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.893795 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-openshift-ca\") pod \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " Nov 02 21:10:45.894153 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.894131 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-openshift-ca" pod "b0b83fd7-15a0-4f8d-b144-7ef56559e77f" (UID: "b0b83fd7-15a0-4f8d-b144-7ef56559e77f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:45.895838 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.895808 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-xks-ca" pod "b0b83fd7-15a0-4f8d-b144-7ef56559e77f" (UID: "b0b83fd7-15a0-4f8d-b144-7ef56559e77f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:45.995065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.995028 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-kube-api-access-p4tlh\" (UniqueName: \"kubernetes.io/projected/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-kube-api-access-p4tlh\") pod \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\" (UID: \"b0b83fd7-15a0-4f8d-b144-7ef56559e77f\") " Nov 02 21:10:45.995571 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.995537 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:45.995723 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.995574 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:45.997355 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:45.997325 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-kube-api-access-p4tlh" pod "b0b83fd7-15a0-4f8d-b144-7ef56559e77f" (UID: "b0b83fd7-15a0-4f8d-b144-7ef56559e77f"). InnerVolumeSpecName "kube-api-access-p4tlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:46.096787 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:46.096736 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tlh\" (UniqueName: \"kubernetes.io/projected/b0b83fd7-15a0-4f8d-b144-7ef56559e77f-kube-api-access-p4tlh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:46.727921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:46.727881 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" event={"ID":"b0b83fd7-15a0-4f8d-b144-7ef56559e77f","Type":"ContainerDied","Data":"b80da1f294d1fc8b2dddf4a733428132699463db0016034b1deaa72aecd9544c"} Nov 02 21:10:46.728109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:46.727927 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc" Nov 02 21:10:46.728109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:46.727933 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80da1f294d1fc8b2dddf4a733428132699463db0016034b1deaa72aecd9544c" Nov 02 21:10:49.173133 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.173094 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz"] Nov 02 21:10:49.173754 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.173736 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b0b83fd7-15a0-4f8d-b144-7ef56559e77f" containerName="revoke-keys" Nov 02 21:10:49.173800 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.173757 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b83fd7-15a0-4f8d-b144-7ef56559e77f" containerName="revoke-keys" Nov 02 21:10:49.173840 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.173800 5800 cpu_manager.go:414] "containerMap: removing container" podUID="83840101-e53a-436f-ae00-bcdd34063062" containerName="payload-processing" Nov 02 21:10:49.173840 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.173806 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83840101-e53a-436f-ae00-bcdd34063062" containerName="payload-processing" Nov 02 21:10:49.173909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.173886 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="83840101-e53a-436f-ae00-bcdd34063062" containerName="payload-processing" Nov 02 21:10:49.173909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.173897 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b83fd7-15a0-4f8d-b144-7ef56559e77f" containerName="revoke-keys" Nov 02 21:10:49.177782 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.177752 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.226591 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.226561 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:10:49.226747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.226610 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:10:49.232100 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.232069 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.232265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.232149 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.232351 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.232315 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9nqv\" (UniqueName: \"kubernetes.io/projected/93e27a80-eadf-4502-83f2-8230b090f7ef-kube-api-access-f9nqv\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.281319 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.281250 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:10:49.281508 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.281358 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:10:49.298114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.288835 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz"] Nov 02 21:10:49.334057 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.334016 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.334238 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.334132 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9nqv\" (UniqueName: \"kubernetes.io/projected/93e27a80-eadf-4502-83f2-8230b090f7ef-kube-api-access-f9nqv\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.334315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.334233 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.334924 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.334901 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.336579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.336557 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.387431 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.387396 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9nqv\" (UniqueName: \"kubernetes.io/projected/93e27a80-eadf-4502-83f2-8230b090f7ef-kube-api-access-f9nqv\") pod \"maas-api-revoke-sub-5da01c77-vz2cz\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.492237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.492152 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:49.870241 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:49.870213 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz"] Nov 02 21:10:49.872465 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:10:49.872428 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e27a80_eadf_4502_83f2_8230b090f7ef.slice/crio-70a4c8a69b3637ae22da40f1940c620f64775e91f39eec36cdf738cb7264c7bd WatchSource:0}: Error finding container 70a4c8a69b3637ae22da40f1940c620f64775e91f39eec36cdf738cb7264c7bd: Status 404 returned error can't find the container with id 70a4c8a69b3637ae22da40f1940c620f64775e91f39eec36cdf738cb7264c7bd Nov 02 21:10:50.804098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:50.804061 5800 generic.go:356] "Generic (PLEG): container finished" podID="93e27a80-eadf-4502-83f2-8230b090f7ef" containerID="010f3aba3e96454c2a5e042b8c7f4713b434f1b1273a9972e4d4421706ddcab2" exitCode=0 Nov 02 21:10:50.804617 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:50.804154 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" event={"ID":"93e27a80-eadf-4502-83f2-8230b090f7ef","Type":"ContainerDied","Data":"010f3aba3e96454c2a5e042b8c7f4713b434f1b1273a9972e4d4421706ddcab2"} Nov 02 21:10:50.804617 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:50.804221 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" event={"ID":"93e27a80-eadf-4502-83f2-8230b090f7ef","Type":"ContainerStarted","Data":"70a4c8a69b3637ae22da40f1940c620f64775e91f39eec36cdf738cb7264c7bd"} Nov 02 21:10:51.835701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:51.835661 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8"] Nov 02 21:10:51.940276 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:51.940227 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st"] Nov 02 21:10:51.982566 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:51.982537 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:52.073925 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.073820 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-xks-ca\") pod \"93e27a80-eadf-4502-83f2-8230b090f7ef\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " Nov 02 21:10:52.074114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.073928 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e27a80-eadf-4502-83f2-8230b090f7ef-kube-api-access-f9nqv\" (UniqueName: \"kubernetes.io/projected/93e27a80-eadf-4502-83f2-8230b090f7ef-kube-api-access-f9nqv\") pod \"93e27a80-eadf-4502-83f2-8230b090f7ef\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " Nov 02 21:10:52.074114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.073957 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-openshift-ca\") pod \"93e27a80-eadf-4502-83f2-8230b090f7ef\" (UID: \"93e27a80-eadf-4502-83f2-8230b090f7ef\") " Nov 02 21:10:52.074449 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.074408 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-openshift-ca" pod "93e27a80-eadf-4502-83f2-8230b090f7ef" (UID: "93e27a80-eadf-4502-83f2-8230b090f7ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:10:52.076139 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.076102 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-xks-ca" pod "93e27a80-eadf-4502-83f2-8230b090f7ef" (UID: "93e27a80-eadf-4502-83f2-8230b090f7ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:10:52.076139 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.076122 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e27a80-eadf-4502-83f2-8230b090f7ef-kube-api-access-f9nqv" pod "93e27a80-eadf-4502-83f2-8230b090f7ef" (UID: "93e27a80-eadf-4502-83f2-8230b090f7ef"). InnerVolumeSpecName "kube-api-access-f9nqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:10:52.175657 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.175612 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:52.175657 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.175657 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9nqv\" (UniqueName: \"kubernetes.io/projected/93e27a80-eadf-4502-83f2-8230b090f7ef-kube-api-access-f9nqv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:52.175869 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.175673 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e27a80-eadf-4502-83f2-8230b090f7ef-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:10:52.696911 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:10:52.696873 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:10:52.819762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.819733 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" Nov 02 21:10:52.819920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.819768 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz" event={"ID":"93e27a80-eadf-4502-83f2-8230b090f7ef","Type":"ContainerDied","Data":"70a4c8a69b3637ae22da40f1940c620f64775e91f39eec36cdf738cb7264c7bd"} Nov 02 21:10:52.819920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:52.819814 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a4c8a69b3637ae22da40f1940c620f64775e91f39eec36cdf738cb7264c7bd" Nov 02 21:10:56.836878 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:56.836798 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" podUID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" containerName="payload-processing" containerID="cri-o://c3a4df3e095e39713f0187f497eee05ba37f261a5e365692f525c8056901cff3" gracePeriod=25 Nov 02 21:10:56.941786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:10:56.941707 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" podUID="190971c5-38b5-4649-98b2-fe1b4b9390c8" containerName="payload-pre-processing" containerID="cri-o://3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c" gracePeriod=25 Nov 02 21:11:03.829885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:03.829797 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" podUID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:11:03.830347 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:03.829807 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" podUID="190971c5-38b5-4649-98b2-fe1b4b9390c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:11:13.829543 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:13.829502 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" podUID="190971c5-38b5-4649-98b2-fe1b4b9390c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:11:13.829942 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:13.829561 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" podUID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:11:16.564995 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.564939 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw"] Nov 02 21:11:16.565726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.565703 5800 cpu_manager.go:414] "containerMap: removing container" podUID="93e27a80-eadf-4502-83f2-8230b090f7ef" containerName="revoke-keys" Nov 02 21:11:16.565832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.565727 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e27a80-eadf-4502-83f2-8230b090f7ef" containerName="revoke-keys" Nov 02 21:11:16.565935 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.565921 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e27a80-eadf-4502-83f2-8230b090f7ef" containerName="revoke-keys" Nov 02 21:11:16.569708 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.569688 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.664856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.664819 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6a4ab0-openshift-default-dockercfg-xzj7t\"" Nov 02 21:11:16.669241 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.669209 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw"] Nov 02 21:11:16.693503 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693456 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.693701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693532 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.693701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693561 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.693845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693703 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.693845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693735 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.693845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693813 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7582fad9-709a-4da9-9331-046f7a15b7e7-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.694014 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693913 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.694014 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.693971 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59f5\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-kube-api-access-r59f5\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.694014 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.694005 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.794734 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794694 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.794734 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794735 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7582fad9-709a-4da9-9331-046f7a15b7e7-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.794963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794782 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.794963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794811 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59f5\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-kube-api-access-r59f5\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.794963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794840 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.794963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794909 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794960 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.794990 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.795066 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795389 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.795263 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-certs\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795389 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.795317 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-credential-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795498 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.795433 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-socket\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795582 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.795561 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-data\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.795795 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.795776 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7582fad9-709a-4da9-9331-046f7a15b7e7-istiod-ca-cert\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.797026 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.797003 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-envoy\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.797528 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.797510 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-podinfo\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.861714 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.861679 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-token\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.864221 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.864194 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59f5\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-kube-api-access-r59f5\") pod \"e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:16.884038 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:16.884013 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:17.012223 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.012159 5800 generic.go:356] "Generic (PLEG): container finished" podID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" containerID="c3a4df3e095e39713f0187f497eee05ba37f261a5e365692f525c8056901cff3" exitCode=0 Nov 02 21:11:17.012439 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.012245 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" event={"ID":"a73e60ab-304a-4bba-93c8-7ae5ab8e7715","Type":"ContainerDied","Data":"c3a4df3e095e39713f0187f497eee05ba37f261a5e365692f525c8056901cff3"} Nov 02 21:11:17.235342 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.235312 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 21:11:17.271929 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.271900 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 21:11:17.315032 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.314998 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw"] Nov 02 21:11:17.315174 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:11:17.315032 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7582fad9_709a_4da9_9331_046f7a15b7e7.slice/crio-c4cf274886545bf2b7af84db78fb5eee6b14e9b06e1f582b5496157fd0aa511c WatchSource:0}: Error finding container c4cf274886545bf2b7af84db78fb5eee6b14e9b06e1f582b5496157fd0aa511c: Status 404 returned error can't find the container with id c4cf274886545bf2b7af84db78fb5eee6b14e9b06e1f582b5496157fd0aa511c Nov 02 21:11:17.318018 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.317984 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:11:17.318122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.318051 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:11:17.318122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.318094 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:11:17.402147 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.402112 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190971c5-38b5-4649-98b2-fe1b4b9390c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190971c5-38b5-4649-98b2-fe1b4b9390c8-plugins-config-volume\") pod \"190971c5-38b5-4649-98b2-fe1b4b9390c8\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " Nov 02 21:11:17.402363 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.402263 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-kube-api-access-4t875\" (UniqueName: \"kubernetes.io/projected/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-kube-api-access-4t875\") pod \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " Nov 02 21:11:17.402454 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.402381 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190971c5-38b5-4649-98b2-fe1b4b9390c8-kube-api-access-mtjh7\" (UniqueName: \"kubernetes.io/projected/190971c5-38b5-4649-98b2-fe1b4b9390c8-kube-api-access-mtjh7\") pod \"190971c5-38b5-4649-98b2-fe1b4b9390c8\" (UID: \"190971c5-38b5-4649-98b2-fe1b4b9390c8\") " Nov 02 21:11:17.402516 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.402464 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-plugins-config-volume\") pod \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\" (UID: \"a73e60ab-304a-4bba-93c8-7ae5ab8e7715\") " Nov 02 21:11:17.402637 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.402593 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190971c5-38b5-4649-98b2-fe1b4b9390c8-plugins-config-volume" pod "190971c5-38b5-4649-98b2-fe1b4b9390c8" (UID: "190971c5-38b5-4649-98b2-fe1b4b9390c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:11:17.402950 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.402922 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-plugins-config-volume" pod "a73e60ab-304a-4bba-93c8-7ae5ab8e7715" (UID: "a73e60ab-304a-4bba-93c8-7ae5ab8e7715"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:11:17.403072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.403057 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:17.403139 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.403074 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190971c5-38b5-4649-98b2-fe1b4b9390c8-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:17.404634 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.404610 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190971c5-38b5-4649-98b2-fe1b4b9390c8-kube-api-access-mtjh7" pod "190971c5-38b5-4649-98b2-fe1b4b9390c8" (UID: "190971c5-38b5-4649-98b2-fe1b4b9390c8"). InnerVolumeSpecName "kube-api-access-mtjh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:17.404832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.404811 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-kube-api-access-4t875" pod "a73e60ab-304a-4bba-93c8-7ae5ab8e7715" (UID: "a73e60ab-304a-4bba-93c8-7ae5ab8e7715"). InnerVolumeSpecName "kube-api-access-4t875". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:17.504628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.503929 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t875\" (UniqueName: \"kubernetes.io/projected/a73e60ab-304a-4bba-93c8-7ae5ab8e7715-kube-api-access-4t875\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:17.504628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:17.503957 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtjh7\" (UniqueName: \"kubernetes.io/projected/190971c5-38b5-4649-98b2-fe1b4b9390c8-kube-api-access-mtjh7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:18.068800 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.068710 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" event={"ID":"7582fad9-709a-4da9-9331-046f7a15b7e7","Type":"ContainerStarted","Data":"2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2"} Nov 02 21:11:18.068800 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.068771 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" event={"ID":"7582fad9-709a-4da9-9331-046f7a15b7e7","Type":"ContainerStarted","Data":"c4cf274886545bf2b7af84db78fb5eee6b14e9b06e1f582b5496157fd0aa511c"} Nov 02 21:11:18.070878 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.070852 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" event={"ID":"a73e60ab-304a-4bba-93c8-7ae5ab8e7715","Type":"ContainerDied","Data":"67124d2d1fecc59c73d38c8377c0f56ccc9cbe85e3b159a9dd61af63406c1a04"} Nov 02 21:11:18.070992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.070902 5800 scope.go:122] "RemoveContainer" containerID="c3a4df3e095e39713f0187f497eee05ba37f261a5e365692f525c8056901cff3" Nov 02 21:11:18.071041 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.071009 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8" Nov 02 21:11:18.074021 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.073490 5800 generic.go:356] "Generic (PLEG): container finished" podID="190971c5-38b5-4649-98b2-fe1b4b9390c8" containerID="3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c" exitCode=0 Nov 02 21:11:18.074021 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.073591 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" Nov 02 21:11:18.074021 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.073611 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" event={"ID":"190971c5-38b5-4649-98b2-fe1b4b9390c8","Type":"ContainerDied","Data":"3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c"} Nov 02 21:11:18.074021 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.073652 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st" event={"ID":"190971c5-38b5-4649-98b2-fe1b4b9390c8","Type":"ContainerDied","Data":"931355431dd3ea7327035844565467776ca8b7328afd604adb41c9da52113012"} Nov 02 21:11:18.084297 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.084256 5800 scope.go:122] "RemoveContainer" containerID="3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c" Nov 02 21:11:18.095313 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.095267 5800 scope.go:122] "RemoveContainer" containerID="3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c" Nov 02 21:11:18.095671 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:11:18.095653 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c\": container with ID starting with 3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c not found: ID does not exist" containerID="3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c" Nov 02 21:11:18.095734 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.095680 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c"} err="failed to get container status \"3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c\": rpc error: code = NotFound desc = could not find container \"3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c\": container with ID starting with 3db18c3164fb79a9c21501e88c7156af949946c54b906c32bcfae09a66e9fd9c not found: ID does not exist" Nov 02 21:11:18.182927 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.182873 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" podStartSLOduration=2.18285989 podStartE2EDuration="2.18285989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:11:18.180020465 +0000 UTC m=+4976.696793049" watchObservedRunningTime="2026-11-02 21:11:18.18285989 +0000 UTC m=+4976.699632474" Nov 02 21:11:18.287753 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.287715 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st"] Nov 02 21:11:18.290987 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.290959 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-6a4ab0-6494b9d76c-dx7st"] Nov 02 21:11:18.393254 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.393161 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8"] Nov 02 21:11:18.400414 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.400383 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6a4ab0-7dfbd9844f-528l8"] Nov 02 21:11:18.884324 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.884274 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:18.886340 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.886306 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 02 21:11:18.886500 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:18.886385 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 02 21:11:19.622666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:19.622630 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190971c5-38b5-4649-98b2-fe1b4b9390c8" path="/var/lib/kubelet/pods/190971c5-38b5-4649-98b2-fe1b4b9390c8/volumes" Nov 02 21:11:19.623080 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:19.623049 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" path="/var/lib/kubelet/pods/a73e60ab-304a-4bba-93c8-7ae5ab8e7715/volumes" Nov 02 21:11:19.884843 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:19.884766 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 02 21:11:19.885005 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:19.884846 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 02 21:11:20.884859 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:20.884821 5800 patch_prober.go:29] interesting pod/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body= Nov 02 21:11:20.885330 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:20.884917 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" Nov 02 21:11:21.411798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:21.411756 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw"] Nov 02 21:11:21.412043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:21.412015 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerName="istio-proxy" containerID="cri-o://2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2" gracePeriod=30 Nov 02 21:11:22.698455 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:11:22.698395 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:11:29.896533 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.896503 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:29.978615 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978522 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-envoy\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.978806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978625 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-podinfo\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.978806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978660 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-data\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.978806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978715 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-socket\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.978806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978743 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-certs\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.978806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978794 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-kube-api-access-r59f5\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-kube-api-access-r59f5\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.979087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978880 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7582fad9-709a-4da9-9331-046f7a15b7e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7582fad9-709a-4da9-9331-046f7a15b7e7-istiod-ca-cert\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.979087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978915 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-token\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-token\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.979087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.978986 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-credential-socket\") pod \"7582fad9-709a-4da9-9331-046f7a15b7e7\" (UID: \"7582fad9-709a-4da9-9331-046f7a15b7e7\") " Nov 02 21:11:29.979087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979006 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-socket" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:29.979087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979035 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-certs" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:29.979087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979050 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-data" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:29.979448 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979349 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7582fad9-709a-4da9-9331-046f7a15b7e7-istiod-ca-cert" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:11:29.979448 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979402 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-credential-socket" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:29.979448 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979440 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:29.979618 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979474 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:29.979618 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.979490 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:29.980935 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.980914 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-envoy" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:11:29.981083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.981066 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-token" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:29.981250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.981225 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-podinfo" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:11:29.981370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:29.981251 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-kube-api-access-r59f5" pod "7582fad9-709a-4da9-9331-046f7a15b7e7" (UID: "7582fad9-709a-4da9-9331-046f7a15b7e7"). InnerVolumeSpecName "kube-api-access-r59f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:30.080387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.080349 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:30.080387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.080379 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59f5\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-kube-api-access-r59f5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:30.080387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.080390 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7582fad9-709a-4da9-9331-046f7a15b7e7-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:30.080646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.080399 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:30.080646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.080410 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:30.080646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.080420 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7582fad9-709a-4da9-9331-046f7a15b7e7-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:30.163244 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.163197 5800 generic.go:356] "Generic (PLEG): container finished" podID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerID="2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2" exitCode=0 Nov 02 21:11:30.163244 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.163242 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" Nov 02 21:11:30.163531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.163271 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" event={"ID":"7582fad9-709a-4da9-9331-046f7a15b7e7","Type":"ContainerDied","Data":"2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2"} Nov 02 21:11:30.163531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.163355 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw" event={"ID":"7582fad9-709a-4da9-9331-046f7a15b7e7","Type":"ContainerDied","Data":"c4cf274886545bf2b7af84db78fb5eee6b14e9b06e1f582b5496157fd0aa511c"} Nov 02 21:11:30.163531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.163394 5800 scope.go:122] "RemoveContainer" containerID="2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2" Nov 02 21:11:30.175399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.175376 5800 scope.go:122] "RemoveContainer" containerID="2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2" Nov 02 21:11:30.175684 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:11:30.175664 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2\": container with ID starting with 2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2 not found: ID does not exist" containerID="2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2" Nov 02 21:11:30.175737 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.175691 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2"} err="failed to get container status \"2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2\": rpc error: code = NotFound desc = could not find container \"2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2\": container with ID starting with 2d03576e99c474f83c6b1ab37b2f87e21eb8b7a2a7cd979bb433e0813a1787c2 not found: ID does not exist" Nov 02 21:11:30.327145 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.327104 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw"] Nov 02 21:11:30.332072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:30.332043 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6a4ab0-openshift-default-5c6f48cb9f-7lgkw"] Nov 02 21:11:31.622161 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:31.622125 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" path="/var/lib/kubelet/pods/7582fad9-709a-4da9-9331-046f7a15b7e7/volumes" Nov 02 21:11:46.591017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.590983 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9"] Nov 02 21:11:46.591533 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591518 5800 cpu_manager.go:414] "containerMap: removing container" podUID="190971c5-38b5-4649-98b2-fe1b4b9390c8" containerName="payload-pre-processing" Nov 02 21:11:46.591583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591534 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190971c5-38b5-4649-98b2-fe1b4b9390c8" containerName="payload-pre-processing" Nov 02 21:11:46.591583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591560 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" containerName="payload-processing" Nov 02 21:11:46.591583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591565 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" containerName="payload-processing" Nov 02 21:11:46.591583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591582 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerName="istio-proxy" Nov 02 21:11:46.591768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591588 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerName="istio-proxy" Nov 02 21:11:46.591768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591667 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="190971c5-38b5-4649-98b2-fe1b4b9390c8" containerName="payload-pre-processing" Nov 02 21:11:46.591768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591678 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73e60ab-304a-4bba-93c8-7ae5ab8e7715" containerName="payload-processing" Nov 02 21:11:46.591768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.591686 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7582fad9-709a-4da9-9331-046f7a15b7e7" containerName="istio-proxy" Nov 02 21:11:46.595139 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.595121 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.647648 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.647612 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9"] Nov 02 21:11:46.669053 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.669025 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf8rs\" (UniqueName: \"kubernetes.io/projected/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-kube-api-access-xf8rs\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.669234 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.669100 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.669234 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.669135 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.689893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.689857 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:11:46.689893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.689878 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:11:46.690148 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.690025 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:11:46.692462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.692442 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:11:46.769998 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.769963 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.770180 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.770002 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.770180 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.770129 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf8rs\" (UniqueName: \"kubernetes.io/projected/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-kube-api-access-xf8rs\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.770740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.770715 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.772507 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.772477 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.823817 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.823791 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf8rs\" (UniqueName: \"kubernetes.io/projected/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-kube-api-access-xf8rs\") pod \"maas-api-revoke-sub-59ac3a97-78dp9\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:46.909687 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:46.909608 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:47.277885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:47.277853 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9"] Nov 02 21:11:47.278652 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:11:47.278613 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45ffb9b_117e_48f4_9cba_1e22d49b31b3.slice/crio-99341dc0aa6b37c5828d16d52a48edf8d57dd8afc5a6841740b163108acb40b3 WatchSource:0}: Error finding container 99341dc0aa6b37c5828d16d52a48edf8d57dd8afc5a6841740b163108acb40b3: Status 404 returned error can't find the container with id 99341dc0aa6b37c5828d16d52a48edf8d57dd8afc5a6841740b163108acb40b3 Nov 02 21:11:48.302036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:48.301998 5800 generic.go:356] "Generic (PLEG): container finished" podID="d45ffb9b-117e-48f4-9cba-1e22d49b31b3" containerID="e108a4fdd9444e59d1efb102d8d7c320836c838578b442de1a4009c1b2152812" exitCode=0 Nov 02 21:11:48.302496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:48.302082 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" event={"ID":"d45ffb9b-117e-48f4-9cba-1e22d49b31b3","Type":"ContainerDied","Data":"e108a4fdd9444e59d1efb102d8d7c320836c838578b442de1a4009c1b2152812"} Nov 02 21:11:48.302496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:48.302158 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" event={"ID":"d45ffb9b-117e-48f4-9cba-1e22d49b31b3","Type":"ContainerStarted","Data":"99341dc0aa6b37c5828d16d52a48edf8d57dd8afc5a6841740b163108acb40b3"} Nov 02 21:11:49.459067 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.459038 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:49.603090 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.602988 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-openshift-ca\") pod \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " Nov 02 21:11:49.603290 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.603125 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-xks-ca\") pod \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " Nov 02 21:11:49.603290 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.603240 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-kube-api-access-xf8rs\" (UniqueName: \"kubernetes.io/projected/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-kube-api-access-xf8rs\") pod \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\" (UID: \"d45ffb9b-117e-48f4-9cba-1e22d49b31b3\") " Nov 02 21:11:49.603522 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.603488 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-openshift-ca" pod "d45ffb9b-117e-48f4-9cba-1e22d49b31b3" (UID: "d45ffb9b-117e-48f4-9cba-1e22d49b31b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:11:49.603737 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.603705 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:49.605517 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.605490 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-xks-ca" pod "d45ffb9b-117e-48f4-9cba-1e22d49b31b3" (UID: "d45ffb9b-117e-48f4-9cba-1e22d49b31b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:11:49.605676 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.605656 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-kube-api-access-xf8rs" pod "d45ffb9b-117e-48f4-9cba-1e22d49b31b3" (UID: "d45ffb9b-117e-48f4-9cba-1e22d49b31b3"). InnerVolumeSpecName "kube-api-access-xf8rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:11:49.705023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.704986 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf8rs\" (UniqueName: \"kubernetes.io/projected/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-kube-api-access-xf8rs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:49.705023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:49.705016 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45ffb9b-117e-48f4-9cba-1e22d49b31b3-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:11:50.316523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:50.316491 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" Nov 02 21:11:50.316523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:50.316515 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9" event={"ID":"d45ffb9b-117e-48f4-9cba-1e22d49b31b3","Type":"ContainerDied","Data":"99341dc0aa6b37c5828d16d52a48edf8d57dd8afc5a6841740b163108acb40b3"} Nov 02 21:11:50.316748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:50.316560 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99341dc0aa6b37c5828d16d52a48edf8d57dd8afc5a6841740b163108acb40b3" Nov 02 21:11:52.699356 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:11:52.699271 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:11:54.602726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:54.602688 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg"] Nov 02 21:11:57.373790 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.373756 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt"] Nov 02 21:11:57.374541 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.374521 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d45ffb9b-117e-48f4-9cba-1e22d49b31b3" containerName="revoke-keys" Nov 02 21:11:57.374657 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.374542 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45ffb9b-117e-48f4-9cba-1e22d49b31b3" containerName="revoke-keys" Nov 02 21:11:57.374714 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.374680 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45ffb9b-117e-48f4-9cba-1e22d49b31b3" containerName="revoke-keys" Nov 02 21:11:57.377544 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.377524 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.429610 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.429555 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt"] Nov 02 21:11:57.471561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.471475 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:11:57.471815 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.471796 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:11:57.472674 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.472659 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:11:57.472969 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.472953 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:11:57.483736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.483700 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.483878 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.483859 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.483945 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.483929 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95nwx\" (UniqueName: \"kubernetes.io/projected/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-kube-api-access-95nwx\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.585344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.585300 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.585532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.585355 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95nwx\" (UniqueName: \"kubernetes.io/projected/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-kube-api-access-95nwx\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.585532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.585407 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.586028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.586004 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.587809 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.587785 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.642180 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.642107 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95nwx\" (UniqueName: \"kubernetes.io/projected/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-kube-api-access-95nwx\") pod \"maas-api-revoke-sub-d0179b70-pzgrt\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:57.690655 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:57.690620 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:11:58.060188 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:58.060158 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt"] Nov 02 21:11:58.061452 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:11:58.061424 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e4ed1b_8db9_44e7_804a_468c8a9c0ee5.slice/crio-4fbab3f43f5f238bbfd258e489e8d2ec884a41aad4287cf60ad22373ccde69c2 WatchSource:0}: Error finding container 4fbab3f43f5f238bbfd258e489e8d2ec884a41aad4287cf60ad22373ccde69c2: Status 404 returned error can't find the container with id 4fbab3f43f5f238bbfd258e489e8d2ec884a41aad4287cf60ad22373ccde69c2 Nov 02 21:11:58.374163 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:58.374125 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" event={"ID":"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5","Type":"ContainerStarted","Data":"441f4efacba9dd0f4a28a4c996f2230e4cab48db1dc6ae504dfee1aaecebad96"} Nov 02 21:11:58.374580 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:58.374181 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" event={"ID":"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5","Type":"ContainerStarted","Data":"4fbab3f43f5f238bbfd258e489e8d2ec884a41aad4287cf60ad22373ccde69c2"} Nov 02 21:11:58.481334 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:58.481261 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" podStartSLOduration=1.481244241 podStartE2EDuration="1.481244241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:11:58.479829977 +0000 UTC m=+5016.996602562" watchObservedRunningTime="2026-11-02 21:11:58.481244241 +0000 UTC m=+5016.998016832" Nov 02 21:11:59.382424 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:59.382385 5800 generic.go:356] "Generic (PLEG): container finished" podID="a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" containerID="441f4efacba9dd0f4a28a4c996f2230e4cab48db1dc6ae504dfee1aaecebad96" exitCode=0 Nov 02 21:11:59.382901 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:59.382476 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" event={"ID":"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5","Type":"ContainerDied","Data":"441f4efacba9dd0f4a28a4c996f2230e4cab48db1dc6ae504dfee1aaecebad96"} Nov 02 21:11:59.603157 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:11:59.603109 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" podUID="e7e14938-4991-4f94-ad15-9cf92a2e185d" containerName="payload-pre-processing" containerID="cri-o://1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a" gracePeriod=25 Nov 02 21:12:00.540302 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.540263 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:12:00.721611 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.721535 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-openshift-ca\") pod \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " Nov 02 21:12:00.721771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.721659 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-xks-ca\") pod \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " Nov 02 21:12:00.721771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.721739 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-kube-api-access-95nwx\" (UniqueName: \"kubernetes.io/projected/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-kube-api-access-95nwx\") pod \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\" (UID: \"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5\") " Nov 02 21:12:00.721954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.721933 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-openshift-ca" pod "a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" (UID: "a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:12:00.722269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.722248 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:00.723805 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.723773 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-xks-ca" pod "a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" (UID: "a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:12:00.723965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.723949 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-kube-api-access-95nwx" pod "a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" (UID: "a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5"). InnerVolumeSpecName "kube-api-access-95nwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:12:00.823866 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.823830 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:00.823866 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:00.823861 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95nwx\" (UniqueName: \"kubernetes.io/projected/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5-kube-api-access-95nwx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:01.397323 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:01.397262 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" Nov 02 21:12:01.397524 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:01.397313 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt" event={"ID":"a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5","Type":"ContainerDied","Data":"4fbab3f43f5f238bbfd258e489e8d2ec884a41aad4287cf60ad22373ccde69c2"} Nov 02 21:12:01.397524 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:01.397413 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fbab3f43f5f238bbfd258e489e8d2ec884a41aad4287cf60ad22373ccde69c2" Nov 02 21:12:03.299664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:03.299622 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" podUID="e7e14938-4991-4f94-ad15-9cf92a2e185d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:12:13.299099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:13.299058 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" podUID="e7e14938-4991-4f94-ad15-9cf92a2e185d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:12:19.881720 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:19.881682 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 21:12:19.930265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:19.930224 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e14938-4991-4f94-ad15-9cf92a2e185d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e14938-4991-4f94-ad15-9cf92a2e185d-plugins-config-volume\") pod \"e7e14938-4991-4f94-ad15-9cf92a2e185d\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " Nov 02 21:12:19.930502 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:19.930480 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e14938-4991-4f94-ad15-9cf92a2e185d-kube-api-access-xtnvl\" (UniqueName: \"kubernetes.io/projected/e7e14938-4991-4f94-ad15-9cf92a2e185d-kube-api-access-xtnvl\") pod \"e7e14938-4991-4f94-ad15-9cf92a2e185d\" (UID: \"e7e14938-4991-4f94-ad15-9cf92a2e185d\") " Nov 02 21:12:19.930684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:19.930663 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e14938-4991-4f94-ad15-9cf92a2e185d-plugins-config-volume" pod "e7e14938-4991-4f94-ad15-9cf92a2e185d" (UID: "e7e14938-4991-4f94-ad15-9cf92a2e185d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:12:19.930869 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:19.930854 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7e14938-4991-4f94-ad15-9cf92a2e185d-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:19.932716 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:19.932680 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e14938-4991-4f94-ad15-9cf92a2e185d-kube-api-access-xtnvl" pod "e7e14938-4991-4f94-ad15-9cf92a2e185d" (UID: "e7e14938-4991-4f94-ad15-9cf92a2e185d"). InnerVolumeSpecName "kube-api-access-xtnvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:12:20.031632 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.031592 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtnvl\" (UniqueName: \"kubernetes.io/projected/e7e14938-4991-4f94-ad15-9cf92a2e185d-kube-api-access-xtnvl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:20.529830 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.529791 5800 generic.go:356] "Generic (PLEG): container finished" podID="e7e14938-4991-4f94-ad15-9cf92a2e185d" containerID="1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a" exitCode=0 Nov 02 21:12:20.529830 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.529843 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" Nov 02 21:12:20.530134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.529835 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" event={"ID":"e7e14938-4991-4f94-ad15-9cf92a2e185d","Type":"ContainerDied","Data":"1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a"} Nov 02 21:12:20.530134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.529913 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg" event={"ID":"e7e14938-4991-4f94-ad15-9cf92a2e185d","Type":"ContainerDied","Data":"998498d32e9c7cc0d1b88702b145003a308ae9175f9ca7e6d8775d381d21b907"} Nov 02 21:12:20.530134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.529945 5800 scope.go:122] "RemoveContainer" containerID="1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a" Nov 02 21:12:20.541991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.541964 5800 scope.go:122] "RemoveContainer" containerID="1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a" Nov 02 21:12:20.542333 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:12:20.542312 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a\": container with ID starting with 1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a not found: ID does not exist" containerID="1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a" Nov 02 21:12:20.542396 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.542343 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a"} err="failed to get container status \"1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a\": rpc error: code = NotFound desc = could not find container \"1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a\": container with ID starting with 1be458b838005a4b5e35d3b6f70c8a5dd0b17b3964c29cab8980e716c9a6dd1a not found: ID does not exist" Nov 02 21:12:20.683653 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.683614 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg"] Nov 02 21:12:20.729920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:20.729885 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-a9ed60-6cff69456c-m8ddg"] Nov 02 21:12:21.625340 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:21.625303 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e14938-4991-4f94-ad15-9cf92a2e185d" path="/var/lib/kubelet/pods/e7e14938-4991-4f94-ad15-9cf92a2e185d/volumes" Nov 02 21:12:22.697123 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:12:22.697086 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:12:31.281723 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:31.281641 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79"] Nov 02 21:12:36.282763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:36.282705 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" podUID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" containerName="payload-processing" containerID="cri-o://5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f" gracePeriod=25 Nov 02 21:12:43.637433 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:43.637390 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" podUID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:12:52.695660 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:12:52.695618 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:12:53.638320 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:53.638268 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" podUID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:12:56.549766 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.549699 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 21:12:56.657337 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.657256 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-kube-api-access-dkm62\" (UniqueName: \"kubernetes.io/projected/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-kube-api-access-dkm62\") pod \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " Nov 02 21:12:56.657559 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.657408 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-plugins-config-volume\") pod \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\" (UID: \"5c11fc3a-992e-4371-8e21-523b5fc6b3b2\") " Nov 02 21:12:56.657926 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.657892 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-plugins-config-volume" pod "5c11fc3a-992e-4371-8e21-523b5fc6b3b2" (UID: "5c11fc3a-992e-4371-8e21-523b5fc6b3b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:12:56.660543 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.660515 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-kube-api-access-dkm62" pod "5c11fc3a-992e-4371-8e21-523b5fc6b3b2" (UID: "5c11fc3a-992e-4371-8e21-523b5fc6b3b2"). InnerVolumeSpecName "kube-api-access-dkm62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:12:56.759558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.759518 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkm62\" (UniqueName: \"kubernetes.io/projected/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-kube-api-access-dkm62\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:56.759558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.759557 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c11fc3a-992e-4371-8e21-523b5fc6b3b2-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:12:56.817833 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.817741 5800 generic.go:356] "Generic (PLEG): container finished" podID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" containerID="5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f" exitCode=0 Nov 02 21:12:56.817833 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.817791 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" Nov 02 21:12:56.817833 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.817818 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" event={"ID":"5c11fc3a-992e-4371-8e21-523b5fc6b3b2","Type":"ContainerDied","Data":"5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f"} Nov 02 21:12:56.818118 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.817876 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79" event={"ID":"5c11fc3a-992e-4371-8e21-523b5fc6b3b2","Type":"ContainerDied","Data":"49b52c38797852fe2ffc0b070e6b3b01ace8216be5eb1d46448aa39bc8c0007d"} Nov 02 21:12:56.818118 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.817901 5800 scope.go:122] "RemoveContainer" containerID="5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f" Nov 02 21:12:56.828881 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.828854 5800 scope.go:122] "RemoveContainer" containerID="5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f" Nov 02 21:12:56.829246 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:12:56.829218 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f\": container with ID starting with 5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f not found: ID does not exist" containerID="5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f" Nov 02 21:12:56.829355 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.829259 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f"} err="failed to get container status \"5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f\": rpc error: code = NotFound desc = could not find container \"5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f\": container with ID starting with 5817e51de252faffa6bcc7151ba326b4e2deccc87589b1837222f4480fe9243f not found: ID does not exist" Nov 02 21:12:56.970747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.970714 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79"] Nov 02 21:12:56.974795 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:56.974768 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-578636-7cc468fcbf-dct79"] Nov 02 21:12:57.622632 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:12:57.622590 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" path="/var/lib/kubelet/pods/5c11fc3a-992e-4371-8e21-523b5fc6b3b2/volumes" Nov 02 21:13:12.248954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:12.248915 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp"] Nov 02 21:13:12.249501 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:12.249133 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" containerID="cri-o://1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9" gracePeriod=30 Nov 02 21:13:13.309960 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.309919 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj"] Nov 02 21:13:13.310521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310504 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" containerName="payload-processing" Nov 02 21:13:13.310576 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310522 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" containerName="payload-processing" Nov 02 21:13:13.310576 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310558 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e7e14938-4991-4f94-ad15-9cf92a2e185d" containerName="payload-pre-processing" Nov 02 21:13:13.310576 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310567 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e14938-4991-4f94-ad15-9cf92a2e185d" containerName="payload-pre-processing" Nov 02 21:13:13.310704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310582 5800 cpu_manager.go:414] "containerMap: removing container" podUID="a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" containerName="revoke-keys" Nov 02 21:13:13.310704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310587 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" containerName="revoke-keys" Nov 02 21:13:13.310704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310668 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e14938-4991-4f94-ad15-9cf92a2e185d" containerName="payload-pre-processing" Nov 02 21:13:13.310704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310679 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" containerName="revoke-keys" Nov 02 21:13:13.310704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.310690 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c11fc3a-992e-4371-8e21-523b5fc6b3b2" containerName="payload-processing" Nov 02 21:13:13.314217 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.314201 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.366373 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.366342 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj"] Nov 02 21:13:13.408436 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.408403 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:13:13.408629 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.408440 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:13:13.409324 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.409306 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:13:13.409391 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.409342 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:13:13.443158 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.443124 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.443405 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.443206 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.443405 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.443375 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pnlw\" (UniqueName: \"kubernetes.io/projected/2189fa77-2f5d-49eb-bafc-b30194ee9d41-kube-api-access-7pnlw\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.544275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.544241 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pnlw\" (UniqueName: \"kubernetes.io/projected/2189fa77-2f5d-49eb-bafc-b30194ee9d41-kube-api-access-7pnlw\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.544490 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.544356 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.544490 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.544434 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.545102 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.545077 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.546807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.546783 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.597753 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.597680 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pnlw\" (UniqueName: \"kubernetes.io/projected/2189fa77-2f5d-49eb-bafc-b30194ee9d41-kube-api-access-7pnlw\") pod \"maas-api-revoke-sub-562a6a3f-ptncj\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.627197 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.627169 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:13.999733 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:13.999701 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj"] Nov 02 21:13:13.999972 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:13:13.999945 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2189fa77_2f5d_49eb_bafc_b30194ee9d41.slice/crio-0312963718f4bf4bb179c876f0d413ad5a9b7a1a9d89c0c54556e8d8bef810b0 WatchSource:0}: Error finding container 0312963718f4bf4bb179c876f0d413ad5a9b7a1a9d89c0c54556e8d8bef810b0: Status 404 returned error can't find the container with id 0312963718f4bf4bb179c876f0d413ad5a9b7a1a9d89c0c54556e8d8bef810b0 Nov 02 21:13:14.941774 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:14.941746 5800 generic.go:356] "Generic (PLEG): container finished" podID="2189fa77-2f5d-49eb-bafc-b30194ee9d41" containerID="387ba628a2484c45fc4aefa5642edc70b454add5abe91aff67ff4608044741ec" exitCode=0 Nov 02 21:13:14.942156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:14.941829 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" event={"ID":"2189fa77-2f5d-49eb-bafc-b30194ee9d41","Type":"ContainerDied","Data":"387ba628a2484c45fc4aefa5642edc70b454add5abe91aff67ff4608044741ec"} Nov 02 21:13:14.942156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:14.941897 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" event={"ID":"2189fa77-2f5d-49eb-bafc-b30194ee9d41","Type":"ContainerStarted","Data":"0312963718f4bf4bb179c876f0d413ad5a9b7a1a9d89c0c54556e8d8bef810b0"} Nov 02 21:13:16.110583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.110554 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:16.275978 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.275933 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2189fa77-2f5d-49eb-bafc-b30194ee9d41-kube-api-access-7pnlw\" (UniqueName: \"kubernetes.io/projected/2189fa77-2f5d-49eb-bafc-b30194ee9d41-kube-api-access-7pnlw\") pod \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " Nov 02 21:13:16.276214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.276013 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-xks-ca\") pod \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " Nov 02 21:13:16.276214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.276089 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-openshift-ca\") pod \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\" (UID: \"2189fa77-2f5d-49eb-bafc-b30194ee9d41\") " Nov 02 21:13:16.276498 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.276473 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-openshift-ca" pod "2189fa77-2f5d-49eb-bafc-b30194ee9d41" (UID: "2189fa77-2f5d-49eb-bafc-b30194ee9d41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:16.278208 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.278184 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2189fa77-2f5d-49eb-bafc-b30194ee9d41-kube-api-access-7pnlw" pod "2189fa77-2f5d-49eb-bafc-b30194ee9d41" (UID: "2189fa77-2f5d-49eb-bafc-b30194ee9d41"). InnerVolumeSpecName "kube-api-access-7pnlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:16.278346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.278261 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-xks-ca" pod "2189fa77-2f5d-49eb-bafc-b30194ee9d41" (UID: "2189fa77-2f5d-49eb-bafc-b30194ee9d41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:13:16.377339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.377271 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pnlw\" (UniqueName: \"kubernetes.io/projected/2189fa77-2f5d-49eb-bafc-b30194ee9d41-kube-api-access-7pnlw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:16.377339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.377335 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:16.377339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.377346 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2189fa77-2f5d-49eb-bafc-b30194ee9d41-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:16.958474 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.958437 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" Nov 02 21:13:16.958864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.958469 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj" event={"ID":"2189fa77-2f5d-49eb-bafc-b30194ee9d41","Type":"ContainerDied","Data":"0312963718f4bf4bb179c876f0d413ad5a9b7a1a9d89c0c54556e8d8bef810b0"} Nov 02 21:13:16.958864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:16.958529 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0312963718f4bf4bb179c876f0d413ad5a9b7a1a9d89c0c54556e8d8bef810b0" Nov 02 21:13:17.522137 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.522059 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 21:13:17.589600 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589561 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-socket\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.589796 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589638 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-istio-token\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-istio-token\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.589796 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589679 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32638407-68f5-4686-b8f4-3367e9009481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638407-68f5-4686-b8f4-3367e9009481-istio-podinfo\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.589796 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589717 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-credential-socket\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.589973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589835 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-kube-api-access-d8lv2\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-kube-api-access-d8lv2\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.589973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589866 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-certs\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.589973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589881 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-socket" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.589973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589923 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-envoy\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.589973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589949 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-data\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.590265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.589979 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32638407-68f5-4686-b8f4-3367e9009481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638407-68f5-4686-b8f4-3367e9009481-istiod-ca-cert\") pod \"32638407-68f5-4686-b8f4-3367e9009481\" (UID: \"32638407-68f5-4686-b8f4-3367e9009481\") " Nov 02 21:13:17.590265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590089 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-credential-socket" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.590265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590193 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-certs" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.590489 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590346 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-data" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.590551 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590484 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.590551 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590507 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.590551 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590523 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.590551 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590538 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.590551 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.590489 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32638407-68f5-4686-b8f4-3367e9009481-istiod-ca-cert" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:17.592569 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.592548 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32638407-68f5-4686-b8f4-3367e9009481-istio-podinfo" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:13:17.592636 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.592598 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-envoy" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:17.592636 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.592621 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-kube-api-access-d8lv2" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "kube-api-access-d8lv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:17.592710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.592654 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-istio-token" pod "32638407-68f5-4686-b8f4-3367e9009481" (UID: "32638407-68f5-4686-b8f4-3367e9009481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:17.671146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.671108 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h"] Nov 02 21:13:17.691077 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.691045 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638407-68f5-4686-b8f4-3367e9009481-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.691077 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.691075 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8lv2\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-kube-api-access-d8lv2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.691304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.691088 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638407-68f5-4686-b8f4-3367e9009481-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.691304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.691097 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638407-68f5-4686-b8f4-3367e9009481-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.691304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.691106 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638407-68f5-4686-b8f4-3367e9009481-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:17.745405 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.745360 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5"] Nov 02 21:13:17.847945 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.847843 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd"] Nov 02 21:13:17.968990 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.968949 5800 generic.go:356] "Generic (PLEG): container finished" podID="32638407-68f5-4686-b8f4-3367e9009481" containerID="1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9" exitCode=0 Nov 02 21:13:17.969171 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.969007 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" Nov 02 21:13:17.969171 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.969007 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" event={"ID":"32638407-68f5-4686-b8f4-3367e9009481","Type":"ContainerDied","Data":"1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9"} Nov 02 21:13:17.969171 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.969060 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp" event={"ID":"32638407-68f5-4686-b8f4-3367e9009481","Type":"ContainerDied","Data":"3e28867ccfd284fdfcae6407d629f94acfe86c97c79130e6d95735680baa079a"} Nov 02 21:13:17.969171 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.969091 5800 scope.go:122] "RemoveContainer" containerID="1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9" Nov 02 21:13:17.980354 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.980330 5800 scope.go:122] "RemoveContainer" containerID="d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1" Nov 02 21:13:17.990269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.990244 5800 scope.go:122] "RemoveContainer" containerID="1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9" Nov 02 21:13:17.990635 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:17.990611 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9\": container with ID starting with 1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9 not found: ID does not exist" containerID="1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9" Nov 02 21:13:17.990717 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.990642 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9"} err="failed to get container status \"1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9\": rpc error: code = NotFound desc = could not find container \"1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9\": container with ID starting with 1b743b1efcd8730faff7bc71503cc5334778d2f08d40966e797e9c108bcdd3a9 not found: ID does not exist" Nov 02 21:13:17.990717 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.990661 5800 scope.go:122] "RemoveContainer" containerID="d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1" Nov 02 21:13:17.990921 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:17.990905 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1\": container with ID starting with d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1 not found: ID does not exist" containerID="d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1" Nov 02 21:13:17.990973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:17.990925 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1"} err="failed to get container status \"d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1\": rpc error: code = NotFound desc = could not find container \"d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1\": container with ID starting with d7200108ee8d5c2d36828b0295683201813e423b7ba573174bfd202b7d40e5b1 not found: ID does not exist" Nov 02 21:13:18.108772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:18.108676 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp"] Nov 02 21:13:18.113142 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:18.113108 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-578636-openshift-default-76496d6455-2zphp"] Nov 02 21:13:19.623668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:19.623631 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32638407-68f5-4686-b8f4-3367e9009481" path="/var/lib/kubelet/pods/32638407-68f5-4686-b8f4-3367e9009481/volumes" Nov 02 21:13:22.598291 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:22.598256 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:13:22.603203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:22.603166 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:13:22.696979 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:22.696941 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:13:22.746149 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:22.746101 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" podUID="292e32f3-d4ed-44e6-9573-94c458727f4d" containerName="payload-processing" containerID="cri-o://20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9" gracePeriod=25 Nov 02 21:13:22.848405 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:22.848306 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" podUID="765773e3-7f3a-4716-b29f-074cfda0758a" containerName="payload-pre-processing" containerID="cri-o://d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f" gracePeriod=25 Nov 02 21:13:24.806947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:24.806906 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" podUID="765773e3-7f3a-4716-b29f-074cfda0758a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:13:25.813461 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:25.813422 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" podUID="292e32f3-d4ed-44e6-9573-94c458727f4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:13:29.237534 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.237495 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl"] Nov 02 21:13:29.238052 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238036 5800 cpu_manager.go:414] "containerMap: removing container" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" Nov 02 21:13:29.238106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238055 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" Nov 02 21:13:29.238106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238078 5800 cpu_manager.go:414] "containerMap: removing container" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" Nov 02 21:13:29.238106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238085 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" Nov 02 21:13:29.238106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238097 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2189fa77-2f5d-49eb-bafc-b30194ee9d41" containerName="revoke-keys" Nov 02 21:13:29.238106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238102 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2189fa77-2f5d-49eb-bafc-b30194ee9d41" containerName="revoke-keys" Nov 02 21:13:29.238269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238196 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" Nov 02 21:13:29.238269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238210 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="32638407-68f5-4686-b8f4-3367e9009481" containerName="istio-proxy" Nov 02 21:13:29.238269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.238223 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2189fa77-2f5d-49eb-bafc-b30194ee9d41" containerName="revoke-keys" Nov 02 21:13:29.243341 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.243320 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.292346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.292308 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:13:29.295764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.295734 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl"] Nov 02 21:13:29.327320 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.327264 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.327511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.327458 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.327665 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.327634 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqd6\" (UniqueName: \"kubernetes.io/projected/0f832e7f-806d-47f0-b6bd-8203a42da733-kube-api-access-vcqd6\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.337621 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.337587 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:13:29.338715 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.338692 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:13:29.338842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.338725 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:13:29.428682 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.428646 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqd6\" (UniqueName: \"kubernetes.io/projected/0f832e7f-806d-47f0-b6bd-8203a42da733-kube-api-access-vcqd6\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.428907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.428693 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.428907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.428781 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.429506 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.429469 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.431370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.431342 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.481819 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.481782 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqd6\" (UniqueName: \"kubernetes.io/projected/0f832e7f-806d-47f0-b6bd-8203a42da733-kube-api-access-vcqd6\") pod \"maas-api-revoke-sub-512e10c2-zfjnl\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.560997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.560960 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:29.937774 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:29.937744 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl"] Nov 02 21:13:29.939950 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:13:29.939920 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f832e7f_806d_47f0_b6bd_8203a42da733.slice/crio-069da16f5f2fdf1eaae7f0ddc5949037ba8d141b32e111978e3b082eb4a55888 WatchSource:0}: Error finding container 069da16f5f2fdf1eaae7f0ddc5949037ba8d141b32e111978e3b082eb4a55888: Status 404 returned error can't find the container with id 069da16f5f2fdf1eaae7f0ddc5949037ba8d141b32e111978e3b082eb4a55888 Nov 02 21:13:30.060497 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:30.060456 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" event={"ID":"0f832e7f-806d-47f0-b6bd-8203a42da733","Type":"ContainerStarted","Data":"069da16f5f2fdf1eaae7f0ddc5949037ba8d141b32e111978e3b082eb4a55888"} Nov 02 21:13:30.601997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:30.601955 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g"] Nov 02 21:13:30.708382 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:30.708341 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d"] Nov 02 21:13:31.070573 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:31.070529 5800 generic.go:356] "Generic (PLEG): container finished" podID="0f832e7f-806d-47f0-b6bd-8203a42da733" containerID="827193f21e1d4bd6399c6cff161c9cd6c3ec270d91b1ee173bbc7326da96a2fc" exitCode=0 Nov 02 21:13:31.070766 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:31.070606 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" event={"ID":"0f832e7f-806d-47f0-b6bd-8203a42da733","Type":"ContainerDied","Data":"827193f21e1d4bd6399c6cff161c9cd6c3ec270d91b1ee173bbc7326da96a2fc"} Nov 02 21:13:32.233530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.233498 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:32.366137 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.366024 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-xks-ca\") pod \"0f832e7f-806d-47f0-b6bd-8203a42da733\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " Nov 02 21:13:32.366137 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.366089 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f832e7f-806d-47f0-b6bd-8203a42da733-kube-api-access-vcqd6\" (UniqueName: \"kubernetes.io/projected/0f832e7f-806d-47f0-b6bd-8203a42da733-kube-api-access-vcqd6\") pod \"0f832e7f-806d-47f0-b6bd-8203a42da733\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " Nov 02 21:13:32.366535 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.366150 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-openshift-ca\") pod \"0f832e7f-806d-47f0-b6bd-8203a42da733\" (UID: \"0f832e7f-806d-47f0-b6bd-8203a42da733\") " Nov 02 21:13:32.366779 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.366749 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-openshift-ca" pod "0f832e7f-806d-47f0-b6bd-8203a42da733" (UID: "0f832e7f-806d-47f0-b6bd-8203a42da733"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:32.367245 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.367206 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:32.368547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.368523 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f832e7f-806d-47f0-b6bd-8203a42da733-kube-api-access-vcqd6" pod "0f832e7f-806d-47f0-b6bd-8203a42da733" (UID: "0f832e7f-806d-47f0-b6bd-8203a42da733"). InnerVolumeSpecName "kube-api-access-vcqd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:32.369527 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.369504 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-xks-ca" pod "0f832e7f-806d-47f0-b6bd-8203a42da733" (UID: "0f832e7f-806d-47f0-b6bd-8203a42da733"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:13:32.468453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.468414 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f832e7f-806d-47f0-b6bd-8203a42da733-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:32.468453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.468446 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcqd6\" (UniqueName: \"kubernetes.io/projected/0f832e7f-806d-47f0-b6bd-8203a42da733-kube-api-access-vcqd6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:32.693454 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.692586 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" containerID="cri-o://4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60" gracePeriod=45 Nov 02 21:13:32.707644 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.707598 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": dial tcp 10.132.0.96:8000: connect: connection refused" Nov 02 21:13:32.971135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:32.971103 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 21:13:33.075585 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.075533 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-home\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-home\") pod \"ad34c781-a25a-43ff-9ee2-39c875333cee\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " Nov 02 21:13:33.075585 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.075588 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad34c781-a25a-43ff-9ee2-39c875333cee-kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/ad34c781-a25a-43ff-9ee2-39c875333cee-kube-api-access-ng78h\") pod \"ad34c781-a25a-43ff-9ee2-39c875333cee\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " Nov 02 21:13:33.075893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.075647 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-tmp-dir\") pod \"ad34c781-a25a-43ff-9ee2-39c875333cee\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " Nov 02 21:13:33.075893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.075709 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad34c781-a25a-43ff-9ee2-39c875333cee-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34c781-a25a-43ff-9ee2-39c875333cee-tls-certs\") pod \"ad34c781-a25a-43ff-9ee2-39c875333cee\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " Nov 02 21:13:33.075893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.075779 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-model-cache\") pod \"ad34c781-a25a-43ff-9ee2-39c875333cee\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " Nov 02 21:13:33.075893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.075825 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-home" pod "ad34c781-a25a-43ff-9ee2-39c875333cee" (UID: "ad34c781-a25a-43ff-9ee2-39c875333cee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.075893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.075838 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-dshm\") pod \"ad34c781-a25a-43ff-9ee2-39c875333cee\" (UID: \"ad34c781-a25a-43ff-9ee2-39c875333cee\") " Nov 02 21:13:33.076276 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.076250 5800 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-home\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.076390 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.076321 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-tmp-dir" pod "ad34c781-a25a-43ff-9ee2-39c875333cee" (UID: "ad34c781-a25a-43ff-9ee2-39c875333cee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.076858 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.076793 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-model-cache" pod "ad34c781-a25a-43ff-9ee2-39c875333cee" (UID: "ad34c781-a25a-43ff-9ee2-39c875333cee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.082680 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.082391 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad34c781-a25a-43ff-9ee2-39c875333cee-tls-certs" pod "ad34c781-a25a-43ff-9ee2-39c875333cee" (UID: "ad34c781-a25a-43ff-9ee2-39c875333cee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:13:33.082680 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.082410 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-dshm" pod "ad34c781-a25a-43ff-9ee2-39c875333cee" (UID: "ad34c781-a25a-43ff-9ee2-39c875333cee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:13:33.082680 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.082637 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad34c781-a25a-43ff-9ee2-39c875333cee-kube-api-access-ng78h" pod "ad34c781-a25a-43ff-9ee2-39c875333cee" (UID: "ad34c781-a25a-43ff-9ee2-39c875333cee"). InnerVolumeSpecName "kube-api-access-ng78h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.098568 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.098625 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl" event={"ID":"0f832e7f-806d-47f0-b6bd-8203a42da733","Type":"ContainerDied","Data":"069da16f5f2fdf1eaae7f0ddc5949037ba8d141b32e111978e3b082eb4a55888"} Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.098672 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="069da16f5f2fdf1eaae7f0ddc5949037ba8d141b32e111978e3b082eb4a55888" Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.100465 5800 generic.go:356] "Generic (PLEG): container finished" podID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerID="4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60" exitCode=0 Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.100561 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" event={"ID":"ad34c781-a25a-43ff-9ee2-39c875333cee","Type":"ContainerDied","Data":"4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60"} Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.100582 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" event={"ID":"ad34c781-a25a-43ff-9ee2-39c875333cee","Type":"ContainerDied","Data":"4baac6784e0e32734ceb3b93b000b52b44ecbebfcbb9731c96b48e3c7bad4a1d"} Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.100598 5800 scope.go:122] "RemoveContainer" containerID="4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60" Nov 02 21:13:33.103312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.100704 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h" Nov 02 21:13:33.122424 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.122397 5800 scope.go:122] "RemoveContainer" containerID="4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60" Nov 02 21:13:33.122803 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:33.122777 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60\": container with ID starting with 4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60 not found: ID does not exist" containerID="4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60" Nov 02 21:13:33.122908 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.122812 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60"} err="failed to get container status \"4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60\": rpc error: code = NotFound desc = could not find container \"4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60\": container with ID starting with 4ec661c81d36ef1790f6345dabf4db01f12932808b240b31196dcdfe489e9c60 not found: ID does not exist" Nov 02 21:13:33.177752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.177705 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng78h\" (UniqueName: \"kubernetes.io/projected/ad34c781-a25a-43ff-9ee2-39c875333cee-kube-api-access-ng78h\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.177752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.177752 5800 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-tmp-dir\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.178034 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.177770 5800 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad34c781-a25a-43ff-9ee2-39c875333cee-tls-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.178034 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.177783 5800 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-model-cache\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.178034 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.177797 5800 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad34c781-a25a-43ff-9ee2-39c875333cee-dshm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:33.258693 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.258652 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h"] Nov 02 21:13:33.263183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.263151 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c25349/distinct-w4-c25349-kserve-758df8b555-8lh7h"] Nov 02 21:13:33.621833 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:33.621749 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" path="/var/lib/kubelet/pods/ad34c781-a25a-43ff-9ee2-39c875333cee/volumes" Nov 02 21:13:34.808006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:34.807968 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" podUID="765773e3-7f3a-4716-b29f-074cfda0758a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:13:35.603186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:35.603131 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" podUID="00724ebe-42d2-43b5-b8a9-86c172611d38" containerName="payload-processing" containerID="cri-o://49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596" gracePeriod=25 Nov 02 21:13:35.709655 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:35.709578 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" podUID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" containerName="payload-pre-processing" containerID="cri-o://8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33" gracePeriod=25 Nov 02 21:13:35.814138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:35.813953 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" podUID="292e32f3-d4ed-44e6-9573-94c458727f4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:13:35.874222 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:35.874193 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 21:13:36.008941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.008899 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00724ebe-42d2-43b5-b8a9-86c172611d38-kube-api-access-p5jn9\" (UniqueName: \"kubernetes.io/projected/00724ebe-42d2-43b5-b8a9-86c172611d38-kube-api-access-p5jn9\") pod \"00724ebe-42d2-43b5-b8a9-86c172611d38\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " Nov 02 21:13:36.009156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.008982 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00724ebe-42d2-43b5-b8a9-86c172611d38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00724ebe-42d2-43b5-b8a9-86c172611d38-plugins-config-volume\") pod \"00724ebe-42d2-43b5-b8a9-86c172611d38\" (UID: \"00724ebe-42d2-43b5-b8a9-86c172611d38\") " Nov 02 21:13:36.009441 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.009417 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00724ebe-42d2-43b5-b8a9-86c172611d38-plugins-config-volume" pod "00724ebe-42d2-43b5-b8a9-86c172611d38" (UID: "00724ebe-42d2-43b5-b8a9-86c172611d38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:36.011186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.011156 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00724ebe-42d2-43b5-b8a9-86c172611d38-kube-api-access-p5jn9" pod "00724ebe-42d2-43b5-b8a9-86c172611d38" (UID: "00724ebe-42d2-43b5-b8a9-86c172611d38"). InnerVolumeSpecName "kube-api-access-p5jn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:36.110325 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.110194 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00724ebe-42d2-43b5-b8a9-86c172611d38-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:36.110325 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.110236 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5jn9\" (UniqueName: \"kubernetes.io/projected/00724ebe-42d2-43b5-b8a9-86c172611d38-kube-api-access-p5jn9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:36.123166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.123126 5800 generic.go:356] "Generic (PLEG): container finished" podID="00724ebe-42d2-43b5-b8a9-86c172611d38" containerID="49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596" exitCode=0 Nov 02 21:13:36.123339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.123176 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" Nov 02 21:13:36.123339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.123215 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" event={"ID":"00724ebe-42d2-43b5-b8a9-86c172611d38","Type":"ContainerDied","Data":"49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596"} Nov 02 21:13:36.123339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.123269 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g" event={"ID":"00724ebe-42d2-43b5-b8a9-86c172611d38","Type":"ContainerDied","Data":"35b808f4436841b2a7ef81bc1ba3513ca72e06fa6548b979d85c98d61f3427d4"} Nov 02 21:13:36.123339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.123311 5800 scope.go:122] "RemoveContainer" containerID="49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596" Nov 02 21:13:36.134214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.134179 5800 scope.go:122] "RemoveContainer" containerID="49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596" Nov 02 21:13:36.134559 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:36.134536 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596\": container with ID starting with 49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596 not found: ID does not exist" containerID="49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596" Nov 02 21:13:36.134636 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.134565 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596"} err="failed to get container status \"49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596\": rpc error: code = NotFound desc = could not find container \"49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596\": container with ID starting with 49383a503baf705f15f24850c2bdbb2a3d5a99f0b8596aa10ef716440ca2e596 not found: ID does not exist" Nov 02 21:13:36.277728 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.277685 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g"] Nov 02 21:13:36.283844 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:36.283813 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c25349-84bb9468f5-qbk4g"] Nov 02 21:13:37.624090 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:37.624052 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00724ebe-42d2-43b5-b8a9-86c172611d38" path="/var/lib/kubelet/pods/00724ebe-42d2-43b5-b8a9-86c172611d38/volumes" Nov 02 21:13:41.897451 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:41.897414 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" podUID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:13:43.016500 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.016429 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 21:13:43.119252 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.119227 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 21:13:43.177808 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.177751 5800 generic.go:356] "Generic (PLEG): container finished" podID="765773e3-7f3a-4716-b29f-074cfda0758a" containerID="d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f" exitCode=0 Nov 02 21:13:43.177808 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.177800 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" Nov 02 21:13:43.178065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.177842 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" event={"ID":"765773e3-7f3a-4716-b29f-074cfda0758a","Type":"ContainerDied","Data":"d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f"} Nov 02 21:13:43.178065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.177912 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd" event={"ID":"765773e3-7f3a-4716-b29f-074cfda0758a","Type":"ContainerDied","Data":"a5d82ed55a831b9d10d2cd5e3eb0dca2e3e8486672d3c46890b3f72565e94853"} Nov 02 21:13:43.178065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.177934 5800 scope.go:122] "RemoveContainer" containerID="d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f" Nov 02 21:13:43.179557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.179498 5800 generic.go:356] "Generic (PLEG): container finished" podID="292e32f3-d4ed-44e6-9573-94c458727f4d" containerID="20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9" exitCode=0 Nov 02 21:13:43.179557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.179537 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" event={"ID":"292e32f3-d4ed-44e6-9573-94c458727f4d","Type":"ContainerDied","Data":"20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9"} Nov 02 21:13:43.179747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.179549 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" Nov 02 21:13:43.179747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.179594 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5" event={"ID":"292e32f3-d4ed-44e6-9573-94c458727f4d","Type":"ContainerDied","Data":"8ab0e437bf4ebcb8d0dc7b0776bcc52598efb2bc78f25914abea31879b20386a"} Nov 02 21:13:43.181885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.181857 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292e32f3-d4ed-44e6-9573-94c458727f4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e32f3-d4ed-44e6-9573-94c458727f4d-plugins-config-volume\") pod \"292e32f3-d4ed-44e6-9573-94c458727f4d\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " Nov 02 21:13:43.182100 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.182082 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292e32f3-d4ed-44e6-9573-94c458727f4d-kube-api-access-bc4kk\" (UniqueName: \"kubernetes.io/projected/292e32f3-d4ed-44e6-9573-94c458727f4d-kube-api-access-bc4kk\") pod \"292e32f3-d4ed-44e6-9573-94c458727f4d\" (UID: \"292e32f3-d4ed-44e6-9573-94c458727f4d\") " Nov 02 21:13:43.182380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.182353 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292e32f3-d4ed-44e6-9573-94c458727f4d-plugins-config-volume" pod "292e32f3-d4ed-44e6-9573-94c458727f4d" (UID: "292e32f3-d4ed-44e6-9573-94c458727f4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:43.182513 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.182498 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/292e32f3-d4ed-44e6-9573-94c458727f4d-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:43.184391 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.184357 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292e32f3-d4ed-44e6-9573-94c458727f4d-kube-api-access-bc4kk" pod "292e32f3-d4ed-44e6-9573-94c458727f4d" (UID: "292e32f3-d4ed-44e6-9573-94c458727f4d"). InnerVolumeSpecName "kube-api-access-bc4kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:43.189331 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.189310 5800 scope.go:122] "RemoveContainer" containerID="d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f" Nov 02 21:13:43.189639 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:43.189616 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f\": container with ID starting with d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f not found: ID does not exist" containerID="d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f" Nov 02 21:13:43.189741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.189645 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f"} err="failed to get container status \"d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f\": rpc error: code = NotFound desc = could not find container \"d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f\": container with ID starting with d3e24b890d52eef92d25baf98477ab700071d1257282b050700dae53e0d1e49f not found: ID does not exist" Nov 02 21:13:43.189741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.189662 5800 scope.go:122] "RemoveContainer" containerID="20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9" Nov 02 21:13:43.201183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.201155 5800 scope.go:122] "RemoveContainer" containerID="20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9" Nov 02 21:13:43.201653 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:43.201627 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9\": container with ID starting with 20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9 not found: ID does not exist" containerID="20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9" Nov 02 21:13:43.201750 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.201663 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9"} err="failed to get container status \"20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9\": rpc error: code = NotFound desc = could not find container \"20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9\": container with ID starting with 20e7ba5128797e78c21fe321be7fccebf72b1def8891f6c7ec352e3985eb54f9 not found: ID does not exist" Nov 02 21:13:43.283740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.283651 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765773e3-7f3a-4716-b29f-074cfda0758a-kube-api-access-2l4t6\" (UniqueName: \"kubernetes.io/projected/765773e3-7f3a-4716-b29f-074cfda0758a-kube-api-access-2l4t6\") pod \"765773e3-7f3a-4716-b29f-074cfda0758a\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " Nov 02 21:13:43.283994 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.283750 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765773e3-7f3a-4716-b29f-074cfda0758a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765773e3-7f3a-4716-b29f-074cfda0758a-plugins-config-volume\") pod \"765773e3-7f3a-4716-b29f-074cfda0758a\" (UID: \"765773e3-7f3a-4716-b29f-074cfda0758a\") " Nov 02 21:13:43.284138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.284109 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765773e3-7f3a-4716-b29f-074cfda0758a-plugins-config-volume" pod "765773e3-7f3a-4716-b29f-074cfda0758a" (UID: "765773e3-7f3a-4716-b29f-074cfda0758a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:43.284265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.284200 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc4kk\" (UniqueName: \"kubernetes.io/projected/292e32f3-d4ed-44e6-9573-94c458727f4d-kube-api-access-bc4kk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:43.284265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.284223 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765773e3-7f3a-4716-b29f-074cfda0758a-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:43.286019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.285997 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765773e3-7f3a-4716-b29f-074cfda0758a-kube-api-access-2l4t6" pod "765773e3-7f3a-4716-b29f-074cfda0758a" (UID: "765773e3-7f3a-4716-b29f-074cfda0758a"). InnerVolumeSpecName "kube-api-access-2l4t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:43.385031 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.384987 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l4t6\" (UniqueName: \"kubernetes.io/projected/765773e3-7f3a-4716-b29f-074cfda0758a-kube-api-access-2l4t6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:43.601394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.598668 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5"] Nov 02 21:13:43.602028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.602001 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-44a484-7c7895ff65-5slw5"] Nov 02 21:13:43.622499 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.622461 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292e32f3-d4ed-44e6-9573-94c458727f4d" path="/var/lib/kubelet/pods/292e32f3-d4ed-44e6-9573-94c458727f4d/volumes" Nov 02 21:13:43.705017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.704983 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd"] Nov 02 21:13:43.710488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:43.710458 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-44a484-7cbbc799f9-zm4wd"] Nov 02 21:13:45.621938 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:45.621903 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765773e3-7f3a-4716-b29f-074cfda0758a" path="/var/lib/kubelet/pods/765773e3-7f3a-4716-b29f-074cfda0758a/volumes" Nov 02 21:13:51.897506 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:51.897466 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" podUID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:13:52.695958 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:52.695912 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:13:55.967923 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:55.967855 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 21:13:56.029557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.029517 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd36395c-5873-43a8-af0a-ab83c08ab4b0-kube-api-access-28m5m\" (UniqueName: \"kubernetes.io/projected/cd36395c-5873-43a8-af0a-ab83c08ab4b0-kube-api-access-28m5m\") pod \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " Nov 02 21:13:56.029771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.029688 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd36395c-5873-43a8-af0a-ab83c08ab4b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd36395c-5873-43a8-af0a-ab83c08ab4b0-plugins-config-volume\") pod \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\" (UID: \"cd36395c-5873-43a8-af0a-ab83c08ab4b0\") " Nov 02 21:13:56.030134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.030106 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd36395c-5873-43a8-af0a-ab83c08ab4b0-plugins-config-volume" pod "cd36395c-5873-43a8-af0a-ab83c08ab4b0" (UID: "cd36395c-5873-43a8-af0a-ab83c08ab4b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:13:56.031657 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.031629 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd36395c-5873-43a8-af0a-ab83c08ab4b0-kube-api-access-28m5m" pod "cd36395c-5873-43a8-af0a-ab83c08ab4b0" (UID: "cd36395c-5873-43a8-af0a-ab83c08ab4b0"). InnerVolumeSpecName "kube-api-access-28m5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:13:56.131807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.131733 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28m5m\" (UniqueName: \"kubernetes.io/projected/cd36395c-5873-43a8-af0a-ab83c08ab4b0-kube-api-access-28m5m\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:56.131807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.131810 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd36395c-5873-43a8-af0a-ab83c08ab4b0-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:13:56.278855 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.278820 5800 generic.go:356] "Generic (PLEG): container finished" podID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" containerID="8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33" exitCode=0 Nov 02 21:13:56.279071 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.278870 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" Nov 02 21:13:56.279071 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.278909 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" event={"ID":"cd36395c-5873-43a8-af0a-ab83c08ab4b0","Type":"ContainerDied","Data":"8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33"} Nov 02 21:13:56.279071 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.278980 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d" event={"ID":"cd36395c-5873-43a8-af0a-ab83c08ab4b0","Type":"ContainerDied","Data":"304fab7e6a1a95e86f2f9d548bba3f829d85b705f28f09a8896c246dc1ca4652"} Nov 02 21:13:56.279071 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.279011 5800 scope.go:122] "RemoveContainer" containerID="8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33" Nov 02 21:13:56.289491 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.289470 5800 scope.go:122] "RemoveContainer" containerID="8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33" Nov 02 21:13:56.289829 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:13:56.289809 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33\": container with ID starting with 8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33 not found: ID does not exist" containerID="8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33" Nov 02 21:13:56.289921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.289835 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33"} err="failed to get container status \"8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33\": rpc error: code = NotFound desc = could not find container \"8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33\": container with ID starting with 8f3a39042ffd9034a5a649994be4de02e285994871d4d264d02545b5cff27f33 not found: ID does not exist" Nov 02 21:13:56.434017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.433950 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d"] Nov 02 21:13:56.443827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:56.439252 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c25349-7df9c58fdc-jkk6d"] Nov 02 21:13:57.623502 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:13:57.623308 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" path="/var/lib/kubelet/pods/cd36395c-5873-43a8-af0a-ab83c08ab4b0/volumes" Nov 02 21:14:06.158473 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:06.158373 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95"] Nov 02 21:14:06.257401 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:06.257360 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj"] Nov 02 21:14:11.159191 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:11.159097 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" podUID="089f26f0-fc0d-4b7a-9238-b597f50aa558" containerName="payload-processing" containerID="cri-o://199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc" gracePeriod=25 Nov 02 21:14:11.258365 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:11.258319 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" podUID="2559bedc-14d4-450e-8e85-8154603e1f57" containerName="payload-pre-processing" containerID="cri-o://0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051" gracePeriod=25 Nov 02 21:14:14.192300 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:14.192245 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" podUID="2559bedc-14d4-450e-8e85-8154603e1f57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:14:14.192731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:14.192248 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" podUID="089f26f0-fc0d-4b7a-9238-b597f50aa558" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:14:22.696491 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:14:22.696451 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:14:24.192841 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:24.192802 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" podUID="2559bedc-14d4-450e-8e85-8154603e1f57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:14:24.192841 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:24.192820 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" podUID="089f26f0-fc0d-4b7a-9238-b597f50aa558" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:14:31.420655 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.420591 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 21:14:31.444861 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.444822 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089f26f0-fc0d-4b7a-9238-b597f50aa558-kube-api-access-nt574\" (UniqueName: \"kubernetes.io/projected/089f26f0-fc0d-4b7a-9238-b597f50aa558-kube-api-access-nt574\") pod \"089f26f0-fc0d-4b7a-9238-b597f50aa558\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " Nov 02 21:14:31.445043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.444964 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089f26f0-fc0d-4b7a-9238-b597f50aa558-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089f26f0-fc0d-4b7a-9238-b597f50aa558-plugins-config-volume\") pod \"089f26f0-fc0d-4b7a-9238-b597f50aa558\" (UID: \"089f26f0-fc0d-4b7a-9238-b597f50aa558\") " Nov 02 21:14:31.445387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.445357 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089f26f0-fc0d-4b7a-9238-b597f50aa558-plugins-config-volume" pod "089f26f0-fc0d-4b7a-9238-b597f50aa558" (UID: "089f26f0-fc0d-4b7a-9238-b597f50aa558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:31.445534 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.445513 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089f26f0-fc0d-4b7a-9238-b597f50aa558-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:31.448034 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.448004 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089f26f0-fc0d-4b7a-9238-b597f50aa558-kube-api-access-nt574" pod "089f26f0-fc0d-4b7a-9238-b597f50aa558" (UID: "089f26f0-fc0d-4b7a-9238-b597f50aa558"). InnerVolumeSpecName "kube-api-access-nt574". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:31.504257 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.504229 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 21:14:31.516292 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.516246 5800 generic.go:356] "Generic (PLEG): container finished" podID="2559bedc-14d4-450e-8e85-8154603e1f57" containerID="0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051" exitCode=0 Nov 02 21:14:31.516485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.516301 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" Nov 02 21:14:31.516485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.516320 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" event={"ID":"2559bedc-14d4-450e-8e85-8154603e1f57","Type":"ContainerDied","Data":"0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051"} Nov 02 21:14:31.516485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.516429 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj" event={"ID":"2559bedc-14d4-450e-8e85-8154603e1f57","Type":"ContainerDied","Data":"510cee138ab416fe833e5ccee3f9682771b63c8942e1402de58bd29bb69075e2"} Nov 02 21:14:31.516485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.516459 5800 scope.go:122] "RemoveContainer" containerID="0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051" Nov 02 21:14:31.517732 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.517707 5800 generic.go:356] "Generic (PLEG): container finished" podID="089f26f0-fc0d-4b7a-9238-b597f50aa558" containerID="199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc" exitCode=0 Nov 02 21:14:31.517856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.517748 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" Nov 02 21:14:31.517856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.517783 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" event={"ID":"089f26f0-fc0d-4b7a-9238-b597f50aa558","Type":"ContainerDied","Data":"199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc"} Nov 02 21:14:31.517856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.517833 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95" event={"ID":"089f26f0-fc0d-4b7a-9238-b597f50aa558","Type":"ContainerDied","Data":"a5af78374b513a4400704c7c9d47cb24a0592b7d40cf5e64f28f3bdf4d4ab71c"} Nov 02 21:14:31.531455 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.531420 5800 scope.go:122] "RemoveContainer" containerID="0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051" Nov 02 21:14:31.531891 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:14:31.531868 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051\": container with ID starting with 0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051 not found: ID does not exist" containerID="0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051" Nov 02 21:14:31.532036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.531897 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051"} err="failed to get container status \"0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051\": rpc error: code = NotFound desc = could not find container \"0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051\": container with ID starting with 0436cbdd9f636f92f94213f5215dd396219fdf8de1948ef5fea070a4bd05b051 not found: ID does not exist" Nov 02 21:14:31.532036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.531917 5800 scope.go:122] "RemoveContainer" containerID="199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc" Nov 02 21:14:31.545082 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.545055 5800 scope.go:122] "RemoveContainer" containerID="199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc" Nov 02 21:14:31.545460 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:14:31.545439 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc\": container with ID starting with 199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc not found: ID does not exist" containerID="199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc" Nov 02 21:14:31.545529 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.545468 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc"} err="failed to get container status \"199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc\": rpc error: code = NotFound desc = could not find container \"199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc\": container with ID starting with 199b38f5e00c48e79c0cef3c1a8251242668f3acd11c05e0b26ed961cf91a1cc not found: ID does not exist" Nov 02 21:14:31.546777 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.546757 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2559bedc-14d4-450e-8e85-8154603e1f57-kube-api-access-4qt2f\" (UniqueName: \"kubernetes.io/projected/2559bedc-14d4-450e-8e85-8154603e1f57-kube-api-access-4qt2f\") pod \"2559bedc-14d4-450e-8e85-8154603e1f57\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " Nov 02 21:14:31.546865 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.546825 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2559bedc-14d4-450e-8e85-8154603e1f57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2559bedc-14d4-450e-8e85-8154603e1f57-plugins-config-volume\") pod \"2559bedc-14d4-450e-8e85-8154603e1f57\" (UID: \"2559bedc-14d4-450e-8e85-8154603e1f57\") " Nov 02 21:14:31.547228 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.547206 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt574\" (UniqueName: \"kubernetes.io/projected/089f26f0-fc0d-4b7a-9238-b597f50aa558-kube-api-access-nt574\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:31.547357 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.547254 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2559bedc-14d4-450e-8e85-8154603e1f57-plugins-config-volume" pod "2559bedc-14d4-450e-8e85-8154603e1f57" (UID: "2559bedc-14d4-450e-8e85-8154603e1f57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:31.549067 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.549042 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2559bedc-14d4-450e-8e85-8154603e1f57-kube-api-access-4qt2f" pod "2559bedc-14d4-450e-8e85-8154603e1f57" (UID: "2559bedc-14d4-450e-8e85-8154603e1f57"). InnerVolumeSpecName "kube-api-access-4qt2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:31.648325 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.648103 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qt2f\" (UniqueName: \"kubernetes.io/projected/2559bedc-14d4-450e-8e85-8154603e1f57-kube-api-access-4qt2f\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:31.648325 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.648138 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2559bedc-14d4-450e-8e85-8154603e1f57-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:31.793505 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.793458 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95"] Nov 02 21:14:31.797389 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.797358 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-09ecd0-6fbf7b6855-pqt95"] Nov 02 21:14:31.924111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.924075 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj"] Nov 02 21:14:31.926825 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:31.926802 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-09ecd0-68c57655b7-gz5nj"] Nov 02 21:14:33.620973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:33.620943 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089f26f0-fc0d-4b7a-9238-b597f50aa558" path="/var/lib/kubelet/pods/089f26f0-fc0d-4b7a-9238-b597f50aa558/volumes" Nov 02 21:14:33.621526 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:33.621436 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2559bedc-14d4-450e-8e85-8154603e1f57" path="/var/lib/kubelet/pods/2559bedc-14d4-450e-8e85-8154603e1f57/volumes" Nov 02 21:14:38.070690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.070656 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs"] Nov 02 21:14:38.071145 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071130 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" containerName="payload-pre-processing" Nov 02 21:14:38.071192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071146 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" containerName="payload-pre-processing" Nov 02 21:14:38.071192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071164 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0f832e7f-806d-47f0-b6bd-8203a42da733" containerName="revoke-keys" Nov 02 21:14:38.071192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071169 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f832e7f-806d-47f0-b6bd-8203a42da733" containerName="revoke-keys" Nov 02 21:14:38.071192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071180 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2559bedc-14d4-450e-8e85-8154603e1f57" containerName="payload-pre-processing" Nov 02 21:14:38.071192 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071188 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2559bedc-14d4-450e-8e85-8154603e1f57" containerName="payload-pre-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071198 5800 cpu_manager.go:414] "containerMap: removing container" podUID="765773e3-7f3a-4716-b29f-074cfda0758a" containerName="payload-pre-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071203 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765773e3-7f3a-4716-b29f-074cfda0758a" containerName="payload-pre-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071225 5800 cpu_manager.go:414] "containerMap: removing container" podUID="089f26f0-fc0d-4b7a-9238-b597f50aa558" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071229 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089f26f0-fc0d-4b7a-9238-b597f50aa558" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071236 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071240 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071245 5800 cpu_manager.go:414] "containerMap: removing container" podUID="292e32f3-d4ed-44e6-9573-94c458727f4d" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071251 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292e32f3-d4ed-44e6-9573-94c458727f4d" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071265 5800 cpu_manager.go:414] "containerMap: removing container" podUID="00724ebe-42d2-43b5-b8a9-86c172611d38" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071272 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00724ebe-42d2-43b5-b8a9-86c172611d38" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071376 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="089f26f0-fc0d-4b7a-9238-b597f50aa558" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071386 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad34c781-a25a-43ff-9ee2-39c875333cee" containerName="main" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071391 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="292e32f3-d4ed-44e6-9573-94c458727f4d" containerName="payload-processing" Nov 02 21:14:38.071399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071400 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="00724ebe-42d2-43b5-b8a9-86c172611d38" containerName="payload-processing" Nov 02 21:14:38.071827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071407 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd36395c-5873-43a8-af0a-ab83c08ab4b0" containerName="payload-pre-processing" Nov 02 21:14:38.071827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071413 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f832e7f-806d-47f0-b6bd-8203a42da733" containerName="revoke-keys" Nov 02 21:14:38.071827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071419 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2559bedc-14d4-450e-8e85-8154603e1f57" containerName="payload-pre-processing" Nov 02 21:14:38.071827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.071425 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="765773e3-7f3a-4716-b29f-074cfda0758a" containerName="payload-pre-processing" Nov 02 21:14:38.074666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.074651 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.122580 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.122542 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc2l9\" (UniqueName: \"kubernetes.io/projected/b74350ac-e8ba-407e-b135-78d8da416b7d-kube-api-access-gc2l9\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.122772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.122710 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.122772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.122760 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.125988 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.125968 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:14:38.126145 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.126124 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:14:38.173010 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.172976 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:14:38.173210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.173140 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:14:38.176265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.176231 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs"] Nov 02 21:14:38.224151 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.224115 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.224409 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.224172 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.224409 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.224215 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc2l9\" (UniqueName: \"kubernetes.io/projected/b74350ac-e8ba-407e-b135-78d8da416b7d-kube-api-access-gc2l9\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.224933 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.224910 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.226866 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.226840 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.278022 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.277978 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc2l9\" (UniqueName: \"kubernetes.io/projected/b74350ac-e8ba-407e-b135-78d8da416b7d-kube-api-access-gc2l9\") pod \"maas-api-revoke-sub-7e023d5a-ghlvs\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.387541 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.387440 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:38.759380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:38.759354 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs"] Nov 02 21:14:38.760618 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:14:38.760594 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74350ac_e8ba_407e_b135_78d8da416b7d.slice/crio-9fc769c19aab09557b57664321f11b266f6ba3212717192697fed7f85855dccc WatchSource:0}: Error finding container 9fc769c19aab09557b57664321f11b266f6ba3212717192697fed7f85855dccc: Status 404 returned error can't find the container with id 9fc769c19aab09557b57664321f11b266f6ba3212717192697fed7f85855dccc Nov 02 21:14:39.576547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:39.576507 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" event={"ID":"b74350ac-e8ba-407e-b135-78d8da416b7d","Type":"ContainerStarted","Data":"ef17446007badd2e7b8438c5239b42f5b16a04718ee5b2777e7662ac7a8e3e79"} Nov 02 21:14:39.576990 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:39.576574 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" event={"ID":"b74350ac-e8ba-407e-b135-78d8da416b7d","Type":"ContainerStarted","Data":"9fc769c19aab09557b57664321f11b266f6ba3212717192697fed7f85855dccc"} Nov 02 21:14:39.683108 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:39.683048 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" podStartSLOduration=1.683005922 podStartE2EDuration="1.683005922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:39.682206025 +0000 UTC m=+5178.198978611" watchObservedRunningTime="2026-11-02 21:14:39.683005922 +0000 UTC m=+5178.199778507" Nov 02 21:14:40.584619 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:40.584585 5800 generic.go:356] "Generic (PLEG): container finished" podID="b74350ac-e8ba-407e-b135-78d8da416b7d" containerID="ef17446007badd2e7b8438c5239b42f5b16a04718ee5b2777e7662ac7a8e3e79" exitCode=0 Nov 02 21:14:40.585033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:40.584672 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" event={"ID":"b74350ac-e8ba-407e-b135-78d8da416b7d","Type":"ContainerDied","Data":"ef17446007badd2e7b8438c5239b42f5b16a04718ee5b2777e7662ac7a8e3e79"} Nov 02 21:14:41.743670 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.743634 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:41.863652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.863574 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-openshift-ca\") pod \"b74350ac-e8ba-407e-b135-78d8da416b7d\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " Nov 02 21:14:41.863652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.863613 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-xks-ca\") pod \"b74350ac-e8ba-407e-b135-78d8da416b7d\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " Nov 02 21:14:41.863911 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.863672 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74350ac-e8ba-407e-b135-78d8da416b7d-kube-api-access-gc2l9\" (UniqueName: \"kubernetes.io/projected/b74350ac-e8ba-407e-b135-78d8da416b7d-kube-api-access-gc2l9\") pod \"b74350ac-e8ba-407e-b135-78d8da416b7d\" (UID: \"b74350ac-e8ba-407e-b135-78d8da416b7d\") " Nov 02 21:14:41.864061 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.864033 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-openshift-ca" pod "b74350ac-e8ba-407e-b135-78d8da416b7d" (UID: "b74350ac-e8ba-407e-b135-78d8da416b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:41.864142 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.864125 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:41.866466 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.866435 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-xks-ca" pod "b74350ac-e8ba-407e-b135-78d8da416b7d" (UID: "b74350ac-e8ba-407e-b135-78d8da416b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:41.866588 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.866549 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74350ac-e8ba-407e-b135-78d8da416b7d-kube-api-access-gc2l9" pod "b74350ac-e8ba-407e-b135-78d8da416b7d" (UID: "b74350ac-e8ba-407e-b135-78d8da416b7d"). InnerVolumeSpecName "kube-api-access-gc2l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:41.965000 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.964962 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b74350ac-e8ba-407e-b135-78d8da416b7d-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:41.965000 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:41.964995 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc2l9\" (UniqueName: \"kubernetes.io/projected/b74350ac-e8ba-407e-b135-78d8da416b7d-kube-api-access-gc2l9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:42.427268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:42.427222 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z"] Nov 02 21:14:42.436994 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:42.436946 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr"] Nov 02 21:14:42.599032 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:42.598990 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" event={"ID":"b74350ac-e8ba-407e-b135-78d8da416b7d","Type":"ContainerDied","Data":"9fc769c19aab09557b57664321f11b266f6ba3212717192697fed7f85855dccc"} Nov 02 21:14:42.599032 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:42.599022 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs" Nov 02 21:14:42.599261 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:42.599084 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc769c19aab09557b57664321f11b266f6ba3212717192697fed7f85855dccc" Nov 02 21:14:50.621119 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.621078 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4"] Nov 02 21:14:50.621607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.621592 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b74350ac-e8ba-407e-b135-78d8da416b7d" containerName="revoke-keys" Nov 02 21:14:50.621650 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.621608 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74350ac-e8ba-407e-b135-78d8da416b7d" containerName="revoke-keys" Nov 02 21:14:50.621695 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.621687 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74350ac-e8ba-407e-b135-78d8da416b7d" containerName="revoke-keys" Nov 02 21:14:50.625141 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.625119 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.674488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.674451 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:14:50.674488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.674477 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:14:50.720954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.720915 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:14:50.721130 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.721047 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:14:50.724115 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.724093 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4"] Nov 02 21:14:50.752562 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.752533 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-888vm\" (UniqueName: \"kubernetes.io/projected/e5508af2-b09c-4efa-875d-1c86f3c23221-kube-api-access-888vm\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.752747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.752600 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.752747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.752732 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.853726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.853690 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-888vm\" (UniqueName: \"kubernetes.io/projected/e5508af2-b09c-4efa-875d-1c86f3c23221-kube-api-access-888vm\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.853952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.853764 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.853952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.853799 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.854553 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.854525 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.856308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.856270 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.907046 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.906976 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-888vm\" (UniqueName: \"kubernetes.io/projected/e5508af2-b09c-4efa-875d-1c86f3c23221-kube-api-access-888vm\") pod \"maas-api-revoke-sub-7b449729-pnsq4\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:50.939644 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:50.939611 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:51.315522 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:51.315486 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4"] Nov 02 21:14:51.317290 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:14:51.317249 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5508af2_b09c_4efa_875d_1c86f3c23221.slice/crio-c26b43431568704ffc46ce1b8eeffd7a51525c1d5c96798ba23958e2709aab6f WatchSource:0}: Error finding container c26b43431568704ffc46ce1b8eeffd7a51525c1d5c96798ba23958e2709aab6f: Status 404 returned error can't find the container with id c26b43431568704ffc46ce1b8eeffd7a51525c1d5c96798ba23958e2709aab6f Nov 02 21:14:51.661086 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:51.661050 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" event={"ID":"e5508af2-b09c-4efa-875d-1c86f3c23221","Type":"ContainerStarted","Data":"8670c6b38e9c1f5f5c269903a9a7fd42318a595f647a9affa627785d2bbd584e"} Nov 02 21:14:51.661500 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:51.661114 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" event={"ID":"e5508af2-b09c-4efa-875d-1c86f3c23221","Type":"ContainerStarted","Data":"c26b43431568704ffc46ce1b8eeffd7a51525c1d5c96798ba23958e2709aab6f"} Nov 02 21:14:51.768978 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:51.768914 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" podStartSLOduration=1.7688994569999998 podStartE2EDuration="1.768899457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:51.765943397 +0000 UTC m=+5190.282715975" watchObservedRunningTime="2026-11-02 21:14:51.768899457 +0000 UTC m=+5190.285672042" Nov 02 21:14:52.669912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:52.669877 5800 generic.go:356] "Generic (PLEG): container finished" podID="e5508af2-b09c-4efa-875d-1c86f3c23221" containerID="8670c6b38e9c1f5f5c269903a9a7fd42318a595f647a9affa627785d2bbd584e" exitCode=0 Nov 02 21:14:52.670352 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:52.669956 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" event={"ID":"e5508af2-b09c-4efa-875d-1c86f3c23221","Type":"ContainerDied","Data":"8670c6b38e9c1f5f5c269903a9a7fd42318a595f647a9affa627785d2bbd584e"} Nov 02 21:14:52.695610 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:14:52.695572 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:14:53.831877 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.831842 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:53.989929 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.989838 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-openshift-ca\") pod \"e5508af2-b09c-4efa-875d-1c86f3c23221\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " Nov 02 21:14:53.990093 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.989929 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-xks-ca\") pod \"e5508af2-b09c-4efa-875d-1c86f3c23221\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " Nov 02 21:14:53.990093 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.989957 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5508af2-b09c-4efa-875d-1c86f3c23221-kube-api-access-888vm\" (UniqueName: \"kubernetes.io/projected/e5508af2-b09c-4efa-875d-1c86f3c23221-kube-api-access-888vm\") pod \"e5508af2-b09c-4efa-875d-1c86f3c23221\" (UID: \"e5508af2-b09c-4efa-875d-1c86f3c23221\") " Nov 02 21:14:53.990296 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.990251 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-openshift-ca" pod "e5508af2-b09c-4efa-875d-1c86f3c23221" (UID: "e5508af2-b09c-4efa-875d-1c86f3c23221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:53.990402 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.990383 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:53.992122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.992097 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-xks-ca" pod "e5508af2-b09c-4efa-875d-1c86f3c23221" (UID: "e5508af2-b09c-4efa-875d-1c86f3c23221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:53.992193 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:53.992176 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5508af2-b09c-4efa-875d-1c86f3c23221-kube-api-access-888vm" pod "e5508af2-b09c-4efa-875d-1c86f3c23221" (UID: "e5508af2-b09c-4efa-875d-1c86f3c23221"). InnerVolumeSpecName "kube-api-access-888vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:54.091727 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:54.091689 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5508af2-b09c-4efa-875d-1c86f3c23221-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:54.091727 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:54.091723 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-888vm\" (UniqueName: \"kubernetes.io/projected/e5508af2-b09c-4efa-875d-1c86f3c23221-kube-api-access-888vm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:54.685752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:54.685717 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" Nov 02 21:14:54.685752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:54.685728 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4" event={"ID":"e5508af2-b09c-4efa-875d-1c86f3c23221","Type":"ContainerDied","Data":"c26b43431568704ffc46ce1b8eeffd7a51525c1d5c96798ba23958e2709aab6f"} Nov 02 21:14:54.685986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:54.685779 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26b43431568704ffc46ce1b8eeffd7a51525c1d5c96798ba23958e2709aab6f" Nov 02 21:14:57.450216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.450135 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" containerID="cri-o://316f12a92a6201495f2160149d8b260d4de665a8ee5c4c5d1962d9f048a5a405" gracePeriod=45 Nov 02 21:14:57.456702 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.456667 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" containerID="cri-o://6ea2e58446cab6105a5639a1bd6a3f91ba16334297009aa57a971670108b4f5e" gracePeriod=45 Nov 02 21:14:57.610706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.610665 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": dial tcp 10.132.0.110:8000: connect: connection refused" Nov 02 21:14:57.710628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.710597 5800 generic.go:356] "Generic (PLEG): container finished" podID="888e258a-9ac1-405c-8055-b00a43205cf8" containerID="6ea2e58446cab6105a5639a1bd6a3f91ba16334297009aa57a971670108b4f5e" exitCode=0 Nov 02 21:14:57.710792 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.710677 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" event={"ID":"888e258a-9ac1-405c-8055-b00a43205cf8","Type":"ContainerDied","Data":"6ea2e58446cab6105a5639a1bd6a3f91ba16334297009aa57a971670108b4f5e"} Nov 02 21:14:57.712615 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.712589 5800 generic.go:356] "Generic (PLEG): container finished" podID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerID="316f12a92a6201495f2160149d8b260d4de665a8ee5c4c5d1962d9f048a5a405" exitCode=0 Nov 02 21:14:57.712740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.712622 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" event={"ID":"75ed7183-0be5-4c22-8795-a4bfedd3d5cd","Type":"ContainerDied","Data":"316f12a92a6201495f2160149d8b260d4de665a8ee5c4c5d1962d9f048a5a405"} Nov 02 21:14:57.712740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.712683 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" event={"ID":"75ed7183-0be5-4c22-8795-a4bfedd3d5cd","Type":"ContainerDied","Data":"f8b44eaf59f92fafa404bedda0bd1b51052ff95c997cd6f69a30ebda0dd106f6"} Nov 02 21:14:57.712740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.712706 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b44eaf59f92fafa404bedda0bd1b51052ff95c997cd6f69a30ebda0dd106f6" Nov 02 21:14:57.728024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.727999 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 21:14:57.760428 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.760405 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 21:14:57.926692 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.926658 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888e258a-9ac1-405c-8055-b00a43205cf8-kube-api-access-gn6qm\" (UniqueName: \"kubernetes.io/projected/888e258a-9ac1-405c-8055-b00a43205cf8-kube-api-access-gn6qm\") pod \"888e258a-9ac1-405c-8055-b00a43205cf8\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " Nov 02 21:14:57.926902 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.926744 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-tmp-dir\") pod \"888e258a-9ac1-405c-8055-b00a43205cf8\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " Nov 02 21:14:57.926902 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.926770 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-home\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-home\") pod \"888e258a-9ac1-405c-8055-b00a43205cf8\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " Nov 02 21:14:57.926902 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.926804 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/888e258a-9ac1-405c-8055-b00a43205cf8-tls-certs\" (UniqueName: \"kubernetes.io/secret/888e258a-9ac1-405c-8055-b00a43205cf8-tls-certs\") pod \"888e258a-9ac1-405c-8055-b00a43205cf8\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " Nov 02 21:14:57.926902 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.926838 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-dshm\") pod \"888e258a-9ac1-405c-8055-b00a43205cf8\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " Nov 02 21:14:57.927140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.926928 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-kube-api-access-hprmk\" (UniqueName: \"kubernetes.io/projected/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-kube-api-access-hprmk\") pod \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " Nov 02 21:14:57.927140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.926961 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tmp-dir\") pod \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " Nov 02 21:14:57.927140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927002 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tls-certs\") pod \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " Nov 02 21:14:57.927140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927035 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-dshm\") pod \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " Nov 02 21:14:57.927140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927092 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-home" pod "888e258a-9ac1-405c-8055-b00a43205cf8" (UID: "888e258a-9ac1-405c-8055-b00a43205cf8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.927140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927104 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-tmp-dir" pod "888e258a-9ac1-405c-8055-b00a43205cf8" (UID: "888e258a-9ac1-405c-8055-b00a43205cf8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.927140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927133 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-home\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-home\") pod \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " Nov 02 21:14:57.927532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927178 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-model-cache\") pod \"888e258a-9ac1-405c-8055-b00a43205cf8\" (UID: \"888e258a-9ac1-405c-8055-b00a43205cf8\") " Nov 02 21:14:57.927532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927220 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-model-cache\") pod \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\" (UID: \"75ed7183-0be5-4c22-8795-a4bfedd3d5cd\") " Nov 02 21:14:57.927532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927272 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tmp-dir" pod "75ed7183-0be5-4c22-8795-a4bfedd3d5cd" (UID: "75ed7183-0be5-4c22-8795-a4bfedd3d5cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.927921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927895 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-model-cache" pod "888e258a-9ac1-405c-8055-b00a43205cf8" (UID: "888e258a-9ac1-405c-8055-b00a43205cf8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.928059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927954 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-model-cache" pod "75ed7183-0be5-4c22-8795-a4bfedd3d5cd" (UID: "75ed7183-0be5-4c22-8795-a4bfedd3d5cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.928059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.927993 5800 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-tmp-dir\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.928059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.928018 5800 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-home\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.928059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.928033 5800 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tmp-dir\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:57.928471 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.928103 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-home" pod "75ed7183-0be5-4c22-8795-a4bfedd3d5cd" (UID: "75ed7183-0be5-4c22-8795-a4bfedd3d5cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.929700 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.929664 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-dshm" pod "888e258a-9ac1-405c-8055-b00a43205cf8" (UID: "888e258a-9ac1-405c-8055-b00a43205cf8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.929824 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.929754 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888e258a-9ac1-405c-8055-b00a43205cf8-tls-certs" pod "888e258a-9ac1-405c-8055-b00a43205cf8" (UID: "888e258a-9ac1-405c-8055-b00a43205cf8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:57.929895 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.929847 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tls-certs" pod "75ed7183-0be5-4c22-8795-a4bfedd3d5cd" (UID: "75ed7183-0be5-4c22-8795-a4bfedd3d5cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:57.930008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.929987 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888e258a-9ac1-405c-8055-b00a43205cf8-kube-api-access-gn6qm" pod "888e258a-9ac1-405c-8055-b00a43205cf8" (UID: "888e258a-9ac1-405c-8055-b00a43205cf8"). InnerVolumeSpecName "kube-api-access-gn6qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:57.930204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.930180 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-dshm" pod "75ed7183-0be5-4c22-8795-a4bfedd3d5cd" (UID: "75ed7183-0be5-4c22-8795-a4bfedd3d5cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:57.930555 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:57.930522 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-kube-api-access-hprmk" pod "75ed7183-0be5-4c22-8795-a4bfedd3d5cd" (UID: "75ed7183-0be5-4c22-8795-a4bfedd3d5cd"). InnerVolumeSpecName "kube-api-access-hprmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:58.029010 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.028974 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hprmk\" (UniqueName: \"kubernetes.io/projected/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-kube-api-access-hprmk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029010 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029008 5800 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-tls-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029024 5800 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-dshm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029035 5800 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-home\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029043 5800 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-model-cache\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029052 5800 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ed7183-0be5-4c22-8795-a4bfedd3d5cd-model-cache\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029061 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6qm\" (UniqueName: \"kubernetes.io/projected/888e258a-9ac1-405c-8055-b00a43205cf8-kube-api-access-gn6qm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029072 5800 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888e258a-9ac1-405c-8055-b00a43205cf8-tls-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.029236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.029080 5800 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888e258a-9ac1-405c-8055-b00a43205cf8-dshm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:58.721156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.721065 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" Nov 02 21:14:58.721156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.721098 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" event={"ID":"888e258a-9ac1-405c-8055-b00a43205cf8","Type":"ContainerDied","Data":"5a9da710f8529532f5bd57c228172583fff0363e3549ad56477592edeefb78ff"} Nov 02 21:14:58.721156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.721160 5800 scope.go:122] "RemoveContainer" containerID="6ea2e58446cab6105a5639a1bd6a3f91ba16334297009aa57a971670108b4f5e" Nov 02 21:14:58.721747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.721334 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z" Nov 02 21:14:58.727121 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.727091 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:14:58.874252 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.874208 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr"] Nov 02 21:14:58.876887 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.876863 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/premium-w1-bbee67-kserve-55d59456c4-dz8zr"] Nov 02 21:14:58.983435 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:58.983334 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z"] Nov 02 21:14:59.012864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:59.012562 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-bbee67/distinct-w1-bbee67-kserve-8485c65785-4d44z"] Nov 02 21:14:59.621652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:59.621617 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" path="/var/lib/kubelet/pods/75ed7183-0be5-4c22-8795-a4bfedd3d5cd/volumes" Nov 02 21:14:59.622080 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:14:59.622061 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" path="/var/lib/kubelet/pods/888e258a-9ac1-405c-8055-b00a43205cf8/volumes" Nov 02 21:15:00.155254 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.155212 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t"] Nov 02 21:15:00.155931 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.155909 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e5508af2-b09c-4efa-875d-1c86f3c23221" containerName="revoke-keys" Nov 02 21:15:00.155931 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.155929 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5508af2-b09c-4efa-875d-1c86f3c23221" containerName="revoke-keys" Nov 02 21:15:00.156143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.155971 5800 cpu_manager.go:414] "containerMap: removing container" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" Nov 02 21:15:00.156143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.155979 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" Nov 02 21:15:00.156143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.155998 5800 cpu_manager.go:414] "containerMap: removing container" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" Nov 02 21:15:00.156143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.156006 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" Nov 02 21:15:00.156143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.156146 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5508af2-b09c-4efa-875d-1c86f3c23221" containerName="revoke-keys" Nov 02 21:15:00.156450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.156163 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ed7183-0be5-4c22-8795-a4bfedd3d5cd" containerName="main" Nov 02 21:15:00.156450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.156175 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="888e258a-9ac1-405c-8055-b00a43205cf8" containerName="main" Nov 02 21:15:00.169131 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.169093 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf"] Nov 02 21:15:00.169312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.169229 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.173672 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.173648 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt"] Nov 02 21:15:00.173817 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.173777 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.189089 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.186912 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.218519 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.218490 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:15:00.218519 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.218490 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:15:00.218728 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.218516 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:15:00.254724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.254688 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxfl\" (UniqueName: \"kubernetes.io/projected/743ae92b-e47a-4ef2-96a4-33f4f762e427-kube-api-access-zcxfl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.255179 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.255154 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8828a332-de30-4a6b-9f9c-730ddd367bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.255401 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.255383 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzcz\" (UniqueName: \"kubernetes.io/projected/8828a332-de30-4a6b-9f9c-730ddd367bac-kube-api-access-nlzcz\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.255631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.255607 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/af388315-f54a-4718-b7bf-ed8cbf7bee6c-kube-api-access-jwwmw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.255829 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.255809 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ae92b-e47a-4ef2-96a4-33f4f762e427-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.255989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.255964 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af388315-f54a-4718-b7bf-ed8cbf7bee6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.256842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.256727 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t"] Nov 02 21:15:00.268686 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.268640 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf"] Nov 02 21:15:00.271664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.271640 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt"] Nov 02 21:15:00.357228 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.357190 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxfl\" (UniqueName: \"kubernetes.io/projected/743ae92b-e47a-4ef2-96a4-33f4f762e427-kube-api-access-zcxfl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.357442 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.357275 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8828a332-de30-4a6b-9f9c-730ddd367bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.357442 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.357325 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzcz\" (UniqueName: \"kubernetes.io/projected/8828a332-de30-4a6b-9f9c-730ddd367bac-kube-api-access-nlzcz\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.357442 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.357370 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/af388315-f54a-4718-b7bf-ed8cbf7bee6c-kube-api-access-jwwmw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.357442 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.357391 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ae92b-e47a-4ef2-96a4-33f4f762e427-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.357442 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.357428 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af388315-f54a-4718-b7bf-ed8cbf7bee6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.358174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.358149 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8828a332-de30-4a6b-9f9c-730ddd367bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.358312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.358194 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af388315-f54a-4718-b7bf-ed8cbf7bee6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.358394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.358347 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ae92b-e47a-4ef2-96a4-33f4f762e427-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.410930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.410854 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzcz\" (UniqueName: \"kubernetes.io/projected/8828a332-de30-4a6b-9f9c-730ddd367bac-kube-api-access-nlzcz\") pod \"maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.410930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.410899 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxfl\" (UniqueName: \"kubernetes.io/projected/743ae92b-e47a-4ef2-96a4-33f4f762e427-kube-api-access-zcxfl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.411134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.410901 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/af388315-f54a-4718-b7bf-ed8cbf7bee6c-kube-api-access-jwwmw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.498616 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.498580 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:00.506922 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.506894 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:00.513215 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.513185 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:00.885096 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.885052 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t"] Nov 02 21:15:00.897438 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:15:00.897405 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8828a332_de30_4a6b_9f9c_730ddd367bac.slice/crio-ce2a74d057cc437fb2dca75bc9cb057cb9b8425ca2ba92e0d88150629daf666d WatchSource:0}: Error finding container ce2a74d057cc437fb2dca75bc9cb057cb9b8425ca2ba92e0d88150629daf666d: Status 404 returned error can't find the container with id ce2a74d057cc437fb2dca75bc9cb057cb9b8425ca2ba92e0d88150629daf666d Nov 02 21:15:00.931235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.931204 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf"] Nov 02 21:15:00.932964 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:15:00.932935 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf388315_f54a_4718_b7bf_ed8cbf7bee6c.slice/crio-edfacfdd35c3e43c4f171bcf13d923b64cd786b81102d24255064bdfa1d650d6 WatchSource:0}: Error finding container edfacfdd35c3e43c4f171bcf13d923b64cd786b81102d24255064bdfa1d650d6: Status 404 returned error can't find the container with id edfacfdd35c3e43c4f171bcf13d923b64cd786b81102d24255064bdfa1d650d6 Nov 02 21:15:00.935641 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:15:00.935582 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743ae92b_e47a_4ef2_96a4_33f4f762e427.slice/crio-cc6b3219ab85166f4c12e34c772f5df99d7dedc423985d74bf68ac6b3ebdd68b WatchSource:0}: Error finding container cc6b3219ab85166f4c12e34c772f5df99d7dedc423985d74bf68ac6b3ebdd68b: Status 404 returned error can't find the container with id cc6b3219ab85166f4c12e34c772f5df99d7dedc423985d74bf68ac6b3ebdd68b Nov 02 21:15:00.936962 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:00.936927 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt"] Nov 02 21:15:01.746458 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.746413 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" event={"ID":"743ae92b-e47a-4ef2-96a4-33f4f762e427","Type":"ContainerStarted","Data":"d98e9aca2eefc4c23c8a69362b7cf3fab287106e712af8a5b43462a3f7744582"} Nov 02 21:15:01.746904 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.746490 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" event={"ID":"743ae92b-e47a-4ef2-96a4-33f4f762e427","Type":"ContainerStarted","Data":"cc6b3219ab85166f4c12e34c772f5df99d7dedc423985d74bf68ac6b3ebdd68b"} Nov 02 21:15:01.748528 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.748497 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" event={"ID":"af388315-f54a-4718-b7bf-ed8cbf7bee6c","Type":"ContainerStarted","Data":"3979746ba50ae6a5c9d75243051604a34cfff9226dd581d717fa17acac38ffe1"} Nov 02 21:15:01.748674 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.748551 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" event={"ID":"af388315-f54a-4718-b7bf-ed8cbf7bee6c","Type":"ContainerStarted","Data":"edfacfdd35c3e43c4f171bcf13d923b64cd786b81102d24255064bdfa1d650d6"} Nov 02 21:15:01.750228 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.750198 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" event={"ID":"8828a332-de30-4a6b-9f9c-730ddd367bac","Type":"ContainerStarted","Data":"ced51c2acbf8c9be8fe90819911138ff357c690d759e4d7ec08ade141f3c51ea"} Nov 02 21:15:01.750395 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.750256 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" event={"ID":"8828a332-de30-4a6b-9f9c-730ddd367bac","Type":"ContainerStarted","Data":"ce2a74d057cc437fb2dca75bc9cb057cb9b8425ca2ba92e0d88150629daf666d"} Nov 02 21:15:01.851455 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.851329 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" podStartSLOduration=1.851310581 podStartE2EDuration="1.851310581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:01.850897474 +0000 UTC m=+5200.367670060" watchObservedRunningTime="2026-11-02 21:15:01.851310581 +0000 UTC m=+5200.368083203" Nov 02 21:15:01.957944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:01.957887 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" podStartSLOduration=1.957870066 podStartE2EDuration="1.957870066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:01.955103456 +0000 UTC m=+5200.471876066" watchObservedRunningTime="2026-11-02 21:15:01.957870066 +0000 UTC m=+5200.474642651" Nov 02 21:15:02.061743 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.061677 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" podStartSLOduration=2.061657856 podStartE2EDuration="2.061657856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:02.059515942 +0000 UTC m=+5200.576288526" watchObservedRunningTime="2026-11-02 21:15:02.061657856 +0000 UTC m=+5200.578430442" Nov 02 21:15:02.757743 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.757707 5800 generic.go:356] "Generic (PLEG): container finished" podID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerID="3979746ba50ae6a5c9d75243051604a34cfff9226dd581d717fa17acac38ffe1" exitCode=60 Nov 02 21:15:02.758178 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.757795 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" event={"ID":"af388315-f54a-4718-b7bf-ed8cbf7bee6c","Type":"ContainerDied","Data":"3979746ba50ae6a5c9d75243051604a34cfff9226dd581d717fa17acac38ffe1"} Nov 02 21:15:02.758178 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.758034 5800 scope.go:122] "RemoveContainer" containerID="3979746ba50ae6a5c9d75243051604a34cfff9226dd581d717fa17acac38ffe1" Nov 02 21:15:02.759623 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.759528 5800 generic.go:356] "Generic (PLEG): container finished" podID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerID="ced51c2acbf8c9be8fe90819911138ff357c690d759e4d7ec08ade141f3c51ea" exitCode=60 Nov 02 21:15:02.759698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.759614 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" event={"ID":"8828a332-de30-4a6b-9f9c-730ddd367bac","Type":"ContainerDied","Data":"ced51c2acbf8c9be8fe90819911138ff357c690d759e4d7ec08ade141f3c51ea"} Nov 02 21:15:02.759837 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.759823 5800 scope.go:122] "RemoveContainer" containerID="ced51c2acbf8c9be8fe90819911138ff357c690d759e4d7ec08ade141f3c51ea" Nov 02 21:15:02.761357 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.761300 5800 generic.go:356] "Generic (PLEG): container finished" podID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerID="d98e9aca2eefc4c23c8a69362b7cf3fab287106e712af8a5b43462a3f7744582" exitCode=60 Nov 02 21:15:02.761427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.761403 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" event={"ID":"743ae92b-e47a-4ef2-96a4-33f4f762e427","Type":"ContainerDied","Data":"d98e9aca2eefc4c23c8a69362b7cf3fab287106e712af8a5b43462a3f7744582"} Nov 02 21:15:02.761602 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:02.761586 5800 scope.go:122] "RemoveContainer" containerID="d98e9aca2eefc4c23c8a69362b7cf3fab287106e712af8a5b43462a3f7744582" Nov 02 21:15:03.769880 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.769838 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" event={"ID":"af388315-f54a-4718-b7bf-ed8cbf7bee6c","Type":"ContainerStarted","Data":"8d47ee58b9014e2b30feb6756ff1c6884911b9eab55956579a13780f489e4867"} Nov 02 21:15:03.771821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.771793 5800 generic.go:356] "Generic (PLEG): container finished" podID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerID="2bbf4994deda4135b5f1322a3a8d1f2b644e868ea1756f9488ca916cc2ed041c" exitCode=60 Nov 02 21:15:03.771947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.771869 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" event={"ID":"8828a332-de30-4a6b-9f9c-730ddd367bac","Type":"ContainerDied","Data":"2bbf4994deda4135b5f1322a3a8d1f2b644e868ea1756f9488ca916cc2ed041c"} Nov 02 21:15:03.771947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.771922 5800 scope.go:122] "RemoveContainer" containerID="ced51c2acbf8c9be8fe90819911138ff357c690d759e4d7ec08ade141f3c51ea" Nov 02 21:15:03.772151 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.772137 5800 scope.go:122] "RemoveContainer" containerID="2bbf4994deda4135b5f1322a3a8d1f2b644e868ea1756f9488ca916cc2ed041c" Nov 02 21:15:03.772429 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:03.772401 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t_odh-ai-gateway-infra(8828a332-de30-4a6b-9f9c-730ddd367bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" Nov 02 21:15:03.774063 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.774040 5800 generic.go:356] "Generic (PLEG): container finished" podID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerID="426739049fefe8ebe5f755ba2102df664f52ab64901f7214fea485f3020d10d9" exitCode=60 Nov 02 21:15:03.774186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.774096 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" event={"ID":"743ae92b-e47a-4ef2-96a4-33f4f762e427","Type":"ContainerDied","Data":"426739049fefe8ebe5f755ba2102df664f52ab64901f7214fea485f3020d10d9"} Nov 02 21:15:03.774387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.774370 5800 scope.go:122] "RemoveContainer" containerID="426739049fefe8ebe5f755ba2102df664f52ab64901f7214fea485f3020d10d9" Nov 02 21:15:03.774693 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:03.774673 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt_odh-ai-gateway-infra(743ae92b-e47a-4ef2-96a4-33f4f762e427)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" Nov 02 21:15:03.806451 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:03.806425 5800 scope.go:122] "RemoveContainer" containerID="d98e9aca2eefc4c23c8a69362b7cf3fab287106e712af8a5b43462a3f7744582" Nov 02 21:15:04.785006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:04.784971 5800 generic.go:356] "Generic (PLEG): container finished" podID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerID="8d47ee58b9014e2b30feb6756ff1c6884911b9eab55956579a13780f489e4867" exitCode=60 Nov 02 21:15:04.785457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:04.785055 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" event={"ID":"af388315-f54a-4718-b7bf-ed8cbf7bee6c","Type":"ContainerDied","Data":"8d47ee58b9014e2b30feb6756ff1c6884911b9eab55956579a13780f489e4867"} Nov 02 21:15:04.785457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:04.785112 5800 scope.go:122] "RemoveContainer" containerID="3979746ba50ae6a5c9d75243051604a34cfff9226dd581d717fa17acac38ffe1" Nov 02 21:15:04.785457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:04.785301 5800 scope.go:122] "RemoveContainer" containerID="8d47ee58b9014e2b30feb6756ff1c6884911b9eab55956579a13780f489e4867" Nov 02 21:15:04.785640 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:04.785623 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf_odh-ai-gateway-infra(af388315-f54a-4718-b7bf-ed8cbf7bee6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" Nov 02 21:15:04.787087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:04.787071 5800 scope.go:122] "RemoveContainer" containerID="2bbf4994deda4135b5f1322a3a8d1f2b644e868ea1756f9488ca916cc2ed041c" Nov 02 21:15:04.787365 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:04.787341 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t_odh-ai-gateway-infra(8828a332-de30-4a6b-9f9c-730ddd367bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" Nov 02 21:15:04.788701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:04.788682 5800 scope.go:122] "RemoveContainer" containerID="426739049fefe8ebe5f755ba2102df664f52ab64901f7214fea485f3020d10d9" Nov 02 21:15:04.788889 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:04.788872 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt_odh-ai-gateway-infra(743ae92b-e47a-4ef2-96a4-33f4f762e427)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" Nov 02 21:15:05.797386 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:05.797355 5800 scope.go:122] "RemoveContainer" containerID="8d47ee58b9014e2b30feb6756ff1c6884911b9eab55956579a13780f489e4867" Nov 02 21:15:05.797813 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:05.797620 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf_odh-ai-gateway-infra(af388315-f54a-4718-b7bf-ed8cbf7bee6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" Nov 02 21:15:10.043555 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:10.043517 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg"] Nov 02 21:15:10.089682 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:10.089644 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439a46e-rlmsg"] Nov 02 21:15:11.621586 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:11.621548 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704b6373-074e-4990-862c-cb99972eefdd" path="/var/lib/kubelet/pods/704b6373-074e-4990-862c-cb99972eefdd/volumes" Nov 02 21:15:13.051607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:13.051573 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247"] Nov 02 21:15:13.098534 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:13.098492 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04e5278-fr247"] Nov 02 21:15:13.621449 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:13.621412 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c97d57e-103a-4f15-b12e-2579cc0e1b88" path="/var/lib/kubelet/pods/9c97d57e-103a-4f15-b12e-2579cc0e1b88/volumes" Nov 02 21:15:14.614381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:14.614344 5800 scope.go:122] "RemoveContainer" containerID="8d47ee58b9014e2b30feb6756ff1c6884911b9eab55956579a13780f489e4867" Nov 02 21:15:14.614931 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:14.614511 5800 scope.go:122] "RemoveContainer" containerID="426739049fefe8ebe5f755ba2102df664f52ab64901f7214fea485f3020d10d9" Nov 02 21:15:15.621551 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.621525 5800 scope.go:122] "RemoveContainer" containerID="2bbf4994deda4135b5f1322a3a8d1f2b644e868ea1756f9488ca916cc2ed041c" Nov 02 21:15:15.864515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.864484 5800 generic.go:356] "Generic (PLEG): container finished" podID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerID="74f4881a118e7c2c01a4ec5c06aa50b24f5d8a73b99c19e6cd629e14ffdafbfe" exitCode=60 Nov 02 21:15:15.864702 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.864526 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" event={"ID":"743ae92b-e47a-4ef2-96a4-33f4f762e427","Type":"ContainerDied","Data":"74f4881a118e7c2c01a4ec5c06aa50b24f5d8a73b99c19e6cd629e14ffdafbfe"} Nov 02 21:15:15.864702 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.864604 5800 scope.go:122] "RemoveContainer" containerID="426739049fefe8ebe5f755ba2102df664f52ab64901f7214fea485f3020d10d9" Nov 02 21:15:15.865062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.865039 5800 scope.go:122] "RemoveContainer" containerID="74f4881a118e7c2c01a4ec5c06aa50b24f5d8a73b99c19e6cd629e14ffdafbfe" Nov 02 21:15:15.865362 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:15.865334 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt_odh-ai-gateway-infra(743ae92b-e47a-4ef2-96a4-33f4f762e427)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" Nov 02 21:15:15.866955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.866932 5800 generic.go:356] "Generic (PLEG): container finished" podID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerID="8786cd1779b81a49e82f494dd363eada2a8089f218b8059bdc4f1f5a553c2437" exitCode=60 Nov 02 21:15:15.867054 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.866996 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" event={"ID":"af388315-f54a-4718-b7bf-ed8cbf7bee6c","Type":"ContainerDied","Data":"8786cd1779b81a49e82f494dd363eada2a8089f218b8059bdc4f1f5a553c2437"} Nov 02 21:15:15.867306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.867274 5800 scope.go:122] "RemoveContainer" containerID="8786cd1779b81a49e82f494dd363eada2a8089f218b8059bdc4f1f5a553c2437" Nov 02 21:15:15.867533 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:15.867515 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf_odh-ai-gateway-infra(af388315-f54a-4718-b7bf-ed8cbf7bee6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" Nov 02 21:15:15.878256 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:15.878226 5800 scope.go:122] "RemoveContainer" containerID="8d47ee58b9014e2b30feb6756ff1c6884911b9eab55956579a13780f489e4867" Nov 02 21:15:16.875859 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:16.875826 5800 generic.go:356] "Generic (PLEG): container finished" podID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerID="c4000a5dbf441cef0a6fa569d5b25f4b385e03f6055fe9f7314f2af27a6d5fb4" exitCode=60 Nov 02 21:15:16.876344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:16.875898 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" event={"ID":"8828a332-de30-4a6b-9f9c-730ddd367bac","Type":"ContainerDied","Data":"c4000a5dbf441cef0a6fa569d5b25f4b385e03f6055fe9f7314f2af27a6d5fb4"} Nov 02 21:15:16.876344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:16.875958 5800 scope.go:122] "RemoveContainer" containerID="2bbf4994deda4135b5f1322a3a8d1f2b644e868ea1756f9488ca916cc2ed041c" Nov 02 21:15:16.876344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:16.876264 5800 scope.go:122] "RemoveContainer" containerID="c4000a5dbf441cef0a6fa569d5b25f4b385e03f6055fe9f7314f2af27a6d5fb4" Nov 02 21:15:16.876601 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:16.876581 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t_odh-ai-gateway-infra(8828a332-de30-4a6b-9f9c-730ddd367bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" Nov 02 21:15:16.980384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:16.980349 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt"] Nov 02 21:15:17.088975 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.088304 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf"] Nov 02 21:15:17.140901 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.140827 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:17.227719 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.227691 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743ae92b-e47a-4ef2-96a4-33f4f762e427-kube-api-access-zcxfl\" (UniqueName: \"kubernetes.io/projected/743ae92b-e47a-4ef2-96a4-33f4f762e427-kube-api-access-zcxfl\") pod \"743ae92b-e47a-4ef2-96a4-33f4f762e427\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " Nov 02 21:15:17.227882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.227839 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743ae92b-e47a-4ef2-96a4-33f4f762e427-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ae92b-e47a-4ef2-96a4-33f4f762e427-maas-api-service-ca\") pod \"743ae92b-e47a-4ef2-96a4-33f4f762e427\" (UID: \"743ae92b-e47a-4ef2-96a4-33f4f762e427\") " Nov 02 21:15:17.228183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.228159 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743ae92b-e47a-4ef2-96a4-33f4f762e427-maas-api-service-ca" pod "743ae92b-e47a-4ef2-96a4-33f4f762e427" (UID: "743ae92b-e47a-4ef2-96a4-33f4f762e427"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:17.229875 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.229854 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743ae92b-e47a-4ef2-96a4-33f4f762e427-kube-api-access-zcxfl" pod "743ae92b-e47a-4ef2-96a4-33f4f762e427" (UID: "743ae92b-e47a-4ef2-96a4-33f4f762e427"). InnerVolumeSpecName "kube-api-access-zcxfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:17.255374 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.255347 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:17.328482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.328452 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af388315-f54a-4718-b7bf-ed8cbf7bee6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af388315-f54a-4718-b7bf-ed8cbf7bee6c-maas-api-service-ca\") pod \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " Nov 02 21:15:17.328681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.328550 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af388315-f54a-4718-b7bf-ed8cbf7bee6c-kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/af388315-f54a-4718-b7bf-ed8cbf7bee6c-kube-api-access-jwwmw\") pod \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\" (UID: \"af388315-f54a-4718-b7bf-ed8cbf7bee6c\") " Nov 02 21:15:17.328828 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.328814 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxfl\" (UniqueName: \"kubernetes.io/projected/743ae92b-e47a-4ef2-96a4-33f4f762e427-kube-api-access-zcxfl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:17.328898 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.328829 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/743ae92b-e47a-4ef2-96a4-33f4f762e427-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:17.328898 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.328857 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af388315-f54a-4718-b7bf-ed8cbf7bee6c-maas-api-service-ca" pod "af388315-f54a-4718-b7bf-ed8cbf7bee6c" (UID: "af388315-f54a-4718-b7bf-ed8cbf7bee6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:17.330725 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.330693 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af388315-f54a-4718-b7bf-ed8cbf7bee6c-kube-api-access-jwwmw" pod "af388315-f54a-4718-b7bf-ed8cbf7bee6c" (UID: "af388315-f54a-4718-b7bf-ed8cbf7bee6c"). InnerVolumeSpecName "kube-api-access-jwwmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:17.430133 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.430035 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af388315-f54a-4718-b7bf-ed8cbf7bee6c-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:17.430133 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.430069 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwwmw\" (UniqueName: \"kubernetes.io/projected/af388315-f54a-4718-b7bf-ed8cbf7bee6c-kube-api-access-jwwmw\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:17.888569 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.888524 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" Nov 02 21:15:17.888569 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.888562 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt" event={"ID":"743ae92b-e47a-4ef2-96a4-33f4f762e427","Type":"ContainerDied","Data":"cc6b3219ab85166f4c12e34c772f5df99d7dedc423985d74bf68ac6b3ebdd68b"} Nov 02 21:15:17.889146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.888620 5800 scope.go:122] "RemoveContainer" containerID="74f4881a118e7c2c01a4ec5c06aa50b24f5d8a73b99c19e6cd629e14ffdafbfe" Nov 02 21:15:17.890305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.890260 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" Nov 02 21:15:17.890432 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.890253 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf" event={"ID":"af388315-f54a-4718-b7bf-ed8cbf7bee6c","Type":"ContainerDied","Data":"edfacfdd35c3e43c4f171bcf13d923b64cd786b81102d24255064bdfa1d650d6"} Nov 02 21:15:17.898664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:17.898641 5800 scope.go:122] "RemoveContainer" containerID="8786cd1779b81a49e82f494dd363eada2a8089f218b8059bdc4f1f5a553c2437" Nov 02 21:15:18.007652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.007619 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t"] Nov 02 21:15:18.050381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.050345 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt"] Nov 02 21:15:18.056502 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.056458 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-76c687-29894235-lwdpt"] Nov 02 21:15:18.164678 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.164637 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf"] Nov 02 21:15:18.167654 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.167623 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-6b12ae-29894235-l8lgf"] Nov 02 21:15:18.181516 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.181485 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:18.237952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.237914 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8828a332-de30-4a6b-9f9c-730ddd367bac-kube-api-access-nlzcz\" (UniqueName: \"kubernetes.io/projected/8828a332-de30-4a6b-9f9c-730ddd367bac-kube-api-access-nlzcz\") pod \"8828a332-de30-4a6b-9f9c-730ddd367bac\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " Nov 02 21:15:18.238157 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.238070 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8828a332-de30-4a6b-9f9c-730ddd367bac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8828a332-de30-4a6b-9f9c-730ddd367bac-maas-api-service-ca\") pod \"8828a332-de30-4a6b-9f9c-730ddd367bac\" (UID: \"8828a332-de30-4a6b-9f9c-730ddd367bac\") " Nov 02 21:15:18.238485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.238461 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8828a332-de30-4a6b-9f9c-730ddd367bac-maas-api-service-ca" pod "8828a332-de30-4a6b-9f9c-730ddd367bac" (UID: "8828a332-de30-4a6b-9f9c-730ddd367bac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:18.240163 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.240138 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8828a332-de30-4a6b-9f9c-730ddd367bac-kube-api-access-nlzcz" pod "8828a332-de30-4a6b-9f9c-730ddd367bac" (UID: "8828a332-de30-4a6b-9f9c-730ddd367bac"). InnerVolumeSpecName "kube-api-access-nlzcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:18.339451 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.339412 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzcz\" (UniqueName: \"kubernetes.io/projected/8828a332-de30-4a6b-9f9c-730ddd367bac-kube-api-access-nlzcz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.339451 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.339445 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8828a332-de30-4a6b-9f9c-730ddd367bac-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:18.898493 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.898454 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" Nov 02 21:15:18.898493 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.898483 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t" event={"ID":"8828a332-de30-4a6b-9f9c-730ddd367bac","Type":"ContainerDied","Data":"ce2a74d057cc437fb2dca75bc9cb057cb9b8425ca2ba92e0d88150629daf666d"} Nov 02 21:15:18.899062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:18.898541 5800 scope.go:122] "RemoveContainer" containerID="c4000a5dbf441cef0a6fa569d5b25f4b385e03f6055fe9f7314f2af27a6d5fb4" Nov 02 21:15:19.051855 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:19.051820 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t"] Nov 02 21:15:19.058254 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:19.058224 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-bbee67-29894235-vwl7t"] Nov 02 21:15:19.622245 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:19.622203 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" path="/var/lib/kubelet/pods/743ae92b-e47a-4ef2-96a4-33f4f762e427/volumes" Nov 02 21:15:19.622749 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:19.622726 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" path="/var/lib/kubelet/pods/8828a332-de30-4a6b-9f9c-730ddd367bac/volumes" Nov 02 21:15:19.623160 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:19.623142 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" path="/var/lib/kubelet/pods/af388315-f54a-4718-b7bf-ed8cbf7bee6c/volumes" Nov 02 21:15:22.043453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.043416 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf"] Nov 02 21:15:22.092018 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.091972 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29913697-xj2zf"] Nov 02 21:15:22.199652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.199617 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v"] Nov 02 21:15:22.200118 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200104 5800 cpu_manager.go:414] "containerMap: removing container" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.200162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200120 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.200162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200138 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:22.200162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200143 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:22.200162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200151 5800 cpu_manager.go:414] "containerMap: removing container" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.200162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200156 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200174 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200180 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200187 5800 cpu_manager.go:414] "containerMap: removing container" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200192 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200205 5800 cpu_manager.go:414] "containerMap: removing container" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200210 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200219 5800 cpu_manager.go:414] "containerMap: removing container" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200224 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200247 5800 cpu_manager.go:414] "containerMap: removing container" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200254 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200339 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.200353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200352 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:22.201083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200362 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:22.201083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200369 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.201083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200375 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:22.201083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200379 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.201083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200385 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:22.201083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.200390 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="af388315-f54a-4718-b7bf-ed8cbf7bee6c" containerName="cleanup" Nov 02 21:15:22.205028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.205011 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.256525 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.256493 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-bbee67-openshift-default-dockercfg-lw7rw\"" Nov 02 21:15:22.258819 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.258785 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v"] Nov 02 21:15:22.280495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280456 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c26695-c909-438e-a744-4136c7183bf5-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.280688 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280525 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.280688 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280624 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.280909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280685 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.280909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280757 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.280909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280835 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4dk\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-kube-api-access-8x4dk\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.280909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280859 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.281143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280951 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c26695-c909-438e-a744-4136c7183bf5-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.281143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.280996 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382426 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382317 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c26695-c909-438e-a744-4136c7183bf5-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382426 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382373 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382461 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382502 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382522 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382565 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4dk\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-kube-api-access-8x4dk\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382598 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382648 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c26695-c909-438e-a744-4136c7183bf5-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.382691 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382677 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.383072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.382991 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-data\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.383138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.383081 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.383138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.383116 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-credential-socket\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.383325 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.383307 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c26695-c909-438e-a744-4136c7183bf5-istiod-ca-cert\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.383429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.383339 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-certs\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.385083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.385058 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-envoy\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.385357 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.385341 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c26695-c909-438e-a744-4136c7183bf5-istio-podinfo\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.436029 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.435991 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-istio-token\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.436413 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.436391 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4dk\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-kube-api-access-8x4dk\") pod \"e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.519310 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:22.519266 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:22.702054 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:22.701963 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:15:23.014120 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:23.014086 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v"] Nov 02 21:15:23.015723 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:15:23.015693 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c26695_c909_438e_a744_4136c7183bf5.slice/crio-1cc187074eb708a27dba9cbaed4f59d2d8a10ff4c635cf8410b1c7772771a9a5 WatchSource:0}: Error finding container 1cc187074eb708a27dba9cbaed4f59d2d8a10ff4c635cf8410b1c7772771a9a5: Status 404 returned error can't find the container with id 1cc187074eb708a27dba9cbaed4f59d2d8a10ff4c635cf8410b1c7772771a9a5 Nov 02 21:15:23.018074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:23.018039 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:15:23.018222 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:23.018120 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:15:23.018222 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:23.018156 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:15:23.625004 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:23.624971 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c2076a-a396-4088-ae79-9df49e9aaa42" path="/var/lib/kubelet/pods/81c2076a-a396-4088-ae79-9df49e9aaa42/volumes" Nov 02 21:15:23.955368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:23.955136 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" event={"ID":"91c26695-c909-438e-a744-4136c7183bf5","Type":"ContainerStarted","Data":"e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071"} Nov 02 21:15:23.955368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:23.955210 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" event={"ID":"91c26695-c909-438e-a744-4136c7183bf5","Type":"ContainerStarted","Data":"1cc187074eb708a27dba9cbaed4f59d2d8a10ff4c635cf8410b1c7772771a9a5"} Nov 02 21:15:24.058499 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:24.058449 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb"] Nov 02 21:15:24.062900 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:24.062868 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c49ea1-vhrwb"] Nov 02 21:15:24.115717 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:24.115659 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" podStartSLOduration=2.115642284 podStartE2EDuration="2.115642284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:24.112997734 +0000 UTC m=+5222.629770317" watchObservedRunningTime="2026-11-02 21:15:24.115642284 +0000 UTC m=+5222.632414869" Nov 02 21:15:24.520144 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:24.520109 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:24.522069 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:24.522039 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 02 21:15:24.522233 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:24.522117 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" podUID="91c26695-c909-438e-a744-4136c7183bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 02 21:15:25.519636 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:25.519599 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 02 21:15:25.520055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:25.519663 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" podUID="91c26695-c909-438e-a744-4136c7183bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 02 21:15:25.622501 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:25.622469 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c86780e-3beb-4412-bc0e-153520c8eaf7" path="/var/lib/kubelet/pods/9c86780e-3beb-4412-bc0e-153520c8eaf7/volumes" Nov 02 21:15:26.421030 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:26.420991 5800 scope.go:122] "RemoveContainer" containerID="316f12a92a6201495f2160149d8b260d4de665a8ee5c4c5d1962d9f048a5a405" Nov 02 21:15:26.520645 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:26.520611 5800 patch_prober.go:29] interesting pod/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 02 21:15:26.521041 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:26.520668 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" podUID="91c26695-c909-438e-a744-4136c7183bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 02 21:15:27.403593 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:27.403553 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v"] Nov 02 21:15:27.403814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:27.403791 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" podUID="91c26695-c909-438e-a744-4136c7183bf5" containerName="istio-proxy" containerID="cri-o://e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071" gracePeriod=30 Nov 02 21:15:29.042708 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:29.042665 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq"] Nov 02 21:15:29.089165 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:29.089125 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b85e70-575dq"] Nov 02 21:15:29.621201 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:29.621164 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536a6fca-b3a8-4fc7-b212-8606774bfd24" path="/var/lib/kubelet/pods/536a6fca-b3a8-4fc7-b212-8606774bfd24/volumes" Nov 02 21:15:34.679466 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.679433 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:34.843924 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.843890 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-credential-socket\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.844126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.843933 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-istio-token\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-istio-token\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.844126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.843965 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-certs\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.844126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844005 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c26695-c909-438e-a744-4136c7183bf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c26695-c909-438e-a744-4136c7183bf5-istiod-ca-cert\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.844126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844082 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91c26695-c909-438e-a744-4136c7183bf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c26695-c909-438e-a744-4136c7183bf5-istio-podinfo\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.844126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844114 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-socket\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.844518 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844144 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-data\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.844518 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844183 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-kube-api-access-8x4dk\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-kube-api-access-8x4dk\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.845684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844644 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-envoy\") pod \"91c26695-c909-438e-a744-4136c7183bf5\" (UID: \"91c26695-c909-438e-a744-4136c7183bf5\") " Nov 02 21:15:34.845684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844211 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-credential-socket" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:34.845684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844580 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-certs" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:34.845684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.844850 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c26695-c909-438e-a744-4136c7183bf5-istiod-ca-cert" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:34.845684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.845042 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-data" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:34.845684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.845210 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-socket" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:34.847008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.846978 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91c26695-c909-438e-a744-4136c7183bf5-istio-podinfo" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:15:34.847627 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.847599 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-envoy" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:15:34.848522 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.848234 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-istio-token" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:34.848837 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.848811 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-kube-api-access-8x4dk" pod "91c26695-c909-438e-a744-4136c7183bf5" (UID: "91c26695-c909-438e-a744-4136c7183bf5"). InnerVolumeSpecName "kube-api-access-8x4dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:34.945939 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.945903 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.945939 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.945942 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.946176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.945958 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.946176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.945973 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c26695-c909-438e-a744-4136c7183bf5-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.946176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.945987 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c26695-c909-438e-a744-4136c7183bf5-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.946176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.945999 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.946176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.946011 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.946176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.946026 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x4dk\" (UniqueName: \"kubernetes.io/projected/91c26695-c909-438e-a744-4136c7183bf5-kube-api-access-8x4dk\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:34.946176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:34.946041 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c26695-c909-438e-a744-4136c7183bf5-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:35.028409 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.028370 5800 generic.go:356] "Generic (PLEG): container finished" podID="91c26695-c909-438e-a744-4136c7183bf5" containerID="e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071" exitCode=0 Nov 02 21:15:35.028620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.028438 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" Nov 02 21:15:35.028620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.028450 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" event={"ID":"91c26695-c909-438e-a744-4136c7183bf5","Type":"ContainerDied","Data":"e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071"} Nov 02 21:15:35.028620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.028524 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v" event={"ID":"91c26695-c909-438e-a744-4136c7183bf5","Type":"ContainerDied","Data":"1cc187074eb708a27dba9cbaed4f59d2d8a10ff4c635cf8410b1c7772771a9a5"} Nov 02 21:15:35.028620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.028554 5800 scope.go:122] "RemoveContainer" containerID="e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071" Nov 02 21:15:35.040434 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.040410 5800 scope.go:122] "RemoveContainer" containerID="e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071" Nov 02 21:15:35.040796 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:35.040773 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071\": container with ID starting with e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071 not found: ID does not exist" containerID="e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071" Nov 02 21:15:35.040903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.040802 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071"} err="failed to get container status \"e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071\": rpc error: code = NotFound desc = could not find container \"e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071\": container with ID starting with e6e0484df0b3c80f407b526cb1c51538ed58c371aa910eddbe43e28735e4b071 not found: ID does not exist" Nov 02 21:15:35.182760 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.182665 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v"] Nov 02 21:15:35.187324 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.187294 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-bbee67-openshift-default-6d56446d9-fqw8v"] Nov 02 21:15:35.622350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:35.622309 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c26695-c909-438e-a744-4136c7183bf5" path="/var/lib/kubelet/pods/91c26695-c909-438e-a744-4136c7183bf5/volumes" Nov 02 21:15:36.044448 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:36.044416 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj"] Nov 02 21:15:36.090762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:36.090716 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec908d9-8wgmj"] Nov 02 21:15:37.620796 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:37.620765 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8245b1a-ba35-4453-882c-562f7ee70e17" path="/var/lib/kubelet/pods/c8245b1a-ba35-4453-882c-562f7ee70e17/volumes" Nov 02 21:15:40.043764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:40.043730 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr"] Nov 02 21:15:40.090781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:40.090750 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7"] Nov 02 21:15:40.093437 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:40.093405 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2189fd46-rvpmr"] Nov 02 21:15:40.095542 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:40.095520 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336b1968-s2gh7"] Nov 02 21:15:41.190951 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.190914 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng"] Nov 02 21:15:41.191484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.191396 5800 cpu_manager.go:414] "containerMap: removing container" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:41.191484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.191410 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8828a332-de30-4a6b-9f9c-730ddd367bac" containerName="cleanup" Nov 02 21:15:41.191484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.191436 5800 cpu_manager.go:414] "containerMap: removing container" podUID="91c26695-c909-438e-a744-4136c7183bf5" containerName="istio-proxy" Nov 02 21:15:41.191484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.191442 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c26695-c909-438e-a744-4136c7183bf5" containerName="istio-proxy" Nov 02 21:15:41.191735 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.191521 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ae92b-e47a-4ef2-96a4-33f4f762e427" containerName="cleanup" Nov 02 21:15:41.191735 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.191533 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c26695-c909-438e-a744-4136c7183bf5" containerName="istio-proxy" Nov 02 21:15:41.196392 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.196358 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.247906 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.247871 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng"] Nov 02 21:15:41.290147 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.290114 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:15:41.290147 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.290140 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:15:41.290381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.290169 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:15:41.294743 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.294725 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:15:41.306350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.306316 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpzx\" (UniqueName: \"kubernetes.io/projected/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-kube-api-access-9bpzx\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.306524 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.306371 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.306598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.306521 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.407728 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.407686 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.407947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.407754 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpzx\" (UniqueName: \"kubernetes.io/projected/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-kube-api-access-9bpzx\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.407947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.407777 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.408558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.408529 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.410221 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.410196 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.461370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.461271 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpzx\" (UniqueName: \"kubernetes.io/projected/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-kube-api-access-9bpzx\") pod \"maas-api-revoke-sub-b0cd95d5-pwdng\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.516451 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.516415 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:41.622916 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.622882 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d7595b-13ba-4348-b5ee-323a5e1e8c44" path="/var/lib/kubelet/pods/47d7595b-13ba-4348-b5ee-323a5e1e8c44/volumes" Nov 02 21:15:41.623333 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.623318 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67819d19-c365-43e4-bc81-1350237bd39c" path="/var/lib/kubelet/pods/67819d19-c365-43e4-bc81-1350237bd39c/volumes" Nov 02 21:15:41.892713 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:41.892686 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng"] Nov 02 21:15:41.894347 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:15:41.894317 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb46e3bb_fd96_4d20_b8f0_7d4dae45e12a.slice/crio-57088a6cf496f37fee8b81bc2bcafc098833af8cba905faa6c531464d25dfaff WatchSource:0}: Error finding container 57088a6cf496f37fee8b81bc2bcafc098833af8cba905faa6c531464d25dfaff: Status 404 returned error can't find the container with id 57088a6cf496f37fee8b81bc2bcafc098833af8cba905faa6c531464d25dfaff Nov 02 21:15:42.079426 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:42.079384 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" event={"ID":"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a","Type":"ContainerStarted","Data":"57088a6cf496f37fee8b81bc2bcafc098833af8cba905faa6c531464d25dfaff"} Nov 02 21:15:43.087772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:43.087737 5800 generic.go:356] "Generic (PLEG): container finished" podID="cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" containerID="bd39124d60a6905f00b87bc60b40f8832708f30713f9adb59525316564a1e1c0" exitCode=0 Nov 02 21:15:43.088168 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:43.087811 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" event={"ID":"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a","Type":"ContainerDied","Data":"bd39124d60a6905f00b87bc60b40f8832708f30713f9adb59525316564a1e1c0"} Nov 02 21:15:44.240521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.240491 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:44.337028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.336985 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-xks-ca\") pod \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " Nov 02 21:15:44.337237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.337039 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-kube-api-access-9bpzx\" (UniqueName: \"kubernetes.io/projected/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-kube-api-access-9bpzx\") pod \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " Nov 02 21:15:44.337237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.337102 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-openshift-ca\") pod \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\" (UID: \"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a\") " Nov 02 21:15:44.337579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.337551 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-openshift-ca" pod "cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" (UID: "cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:44.339275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.339241 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-kube-api-access-9bpzx" pod "cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" (UID: "cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a"). InnerVolumeSpecName "kube-api-access-9bpzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:44.339413 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.339302 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-xks-ca" pod "cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" (UID: "cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:44.438949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.438901 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:44.438949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.438943 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpzx\" (UniqueName: \"kubernetes.io/projected/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-kube-api-access-9bpzx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:44.439219 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:44.438960 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:45.101133 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:45.101100 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" Nov 02 21:15:45.101344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:45.101136 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng" event={"ID":"cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a","Type":"ContainerDied","Data":"57088a6cf496f37fee8b81bc2bcafc098833af8cba905faa6c531464d25dfaff"} Nov 02 21:15:45.101344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:45.101183 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57088a6cf496f37fee8b81bc2bcafc098833af8cba905faa6c531464d25dfaff" Nov 02 21:15:46.049511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:46.049470 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc"] Nov 02 21:15:46.095630 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:46.095596 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62b26cc-qs4dc"] Nov 02 21:15:47.620748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:47.620712 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b83fd7-15a0-4f8d-b144-7ef56559e77f" path="/var/lib/kubelet/pods/b0b83fd7-15a0-4f8d-b144-7ef56559e77f/volumes" Nov 02 21:15:52.699155 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:15:52.699113 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:15:53.024173 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.024137 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk"] Nov 02 21:15:53.024625 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.024611 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" containerName="revoke-keys" Nov 02 21:15:53.024688 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.024627 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" containerName="revoke-keys" Nov 02 21:15:53.024731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.024725 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" containerName="revoke-keys" Nov 02 21:15:53.028566 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.028545 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.042897 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.042866 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz"] Nov 02 21:15:53.070746 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.070708 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da01c77-vz2cz"] Nov 02 21:15:53.087008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.086961 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:15:53.087203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.087049 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:15:53.090255 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.090222 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk"] Nov 02 21:15:53.114521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.114488 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:15:53.114521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.114488 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:15:53.127023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.126983 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjl9\" (UniqueName: \"kubernetes.io/projected/cc1be430-a931-4237-937e-2597f724d57d-kube-api-access-7kjl9\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.127203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.127072 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be430-a931-4237-937e-2597f724d57d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.127203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.127176 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be430-a931-4237-937e-2597f724d57d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.228395 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.228358 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjl9\" (UniqueName: \"kubernetes.io/projected/cc1be430-a931-4237-937e-2597f724d57d-kube-api-access-7kjl9\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.228592 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.228403 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be430-a931-4237-937e-2597f724d57d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.228592 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.228467 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be430-a931-4237-937e-2597f724d57d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.229187 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.229166 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be430-a931-4237-937e-2597f724d57d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.231054 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.231033 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be430-a931-4237-937e-2597f724d57d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.281806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.281723 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjl9\" (UniqueName: \"kubernetes.io/projected/cc1be430-a931-4237-937e-2597f724d57d-kube-api-access-7kjl9\") pod \"maas-api-revoke-sub-cbd82c55-jf9sk\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.342482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.342443 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:53.621170 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.621088 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e27a80-eadf-4502-83f2-8230b090f7ef" path="/var/lib/kubelet/pods/93e27a80-eadf-4502-83f2-8230b090f7ef/volumes" Nov 02 21:15:53.715690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:53.715658 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk"] Nov 02 21:15:53.717475 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:15:53.717446 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1be430_a931_4237_937e_2597f724d57d.slice/crio-0dd3f25b06af7fe679b6a06f029c870fb2f1c53445e970d3ec1cb820045a2892 WatchSource:0}: Error finding container 0dd3f25b06af7fe679b6a06f029c870fb2f1c53445e970d3ec1cb820045a2892: Status 404 returned error can't find the container with id 0dd3f25b06af7fe679b6a06f029c870fb2f1c53445e970d3ec1cb820045a2892 Nov 02 21:15:54.165180 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:54.165138 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" event={"ID":"cc1be430-a931-4237-937e-2597f724d57d","Type":"ContainerStarted","Data":"fc68d40688624900a5a8092b21e917588baa5673637fa313402248f055b08d88"} Nov 02 21:15:54.165409 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:54.165201 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" event={"ID":"cc1be430-a931-4237-937e-2597f724d57d","Type":"ContainerStarted","Data":"0dd3f25b06af7fe679b6a06f029c870fb2f1c53445e970d3ec1cb820045a2892"} Nov 02 21:15:54.272465 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:54.272410 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" podStartSLOduration=2.27239669 podStartE2EDuration="2.27239669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:15:54.270137315 +0000 UTC m=+5252.786909901" watchObservedRunningTime="2026-11-02 21:15:54.27239669 +0000 UTC m=+5252.789169265" Nov 02 21:15:55.175183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:55.175139 5800 generic.go:356] "Generic (PLEG): container finished" podID="cc1be430-a931-4237-937e-2597f724d57d" containerID="fc68d40688624900a5a8092b21e917588baa5673637fa313402248f055b08d88" exitCode=0 Nov 02 21:15:55.175660 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:55.175225 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" event={"ID":"cc1be430-a931-4237-937e-2597f724d57d","Type":"ContainerDied","Data":"fc68d40688624900a5a8092b21e917588baa5673637fa313402248f055b08d88"} Nov 02 21:15:56.334791 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.334761 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:56.461809 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.461727 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1be430-a931-4237-937e-2597f724d57d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be430-a931-4237-937e-2597f724d57d-maas-api-xks-ca\") pod \"cc1be430-a931-4237-937e-2597f724d57d\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " Nov 02 21:15:56.461809 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.461800 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1be430-a931-4237-937e-2597f724d57d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be430-a931-4237-937e-2597f724d57d-maas-api-openshift-ca\") pod \"cc1be430-a931-4237-937e-2597f724d57d\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " Nov 02 21:15:56.462055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.461841 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1be430-a931-4237-937e-2597f724d57d-kube-api-access-7kjl9\" (UniqueName: \"kubernetes.io/projected/cc1be430-a931-4237-937e-2597f724d57d-kube-api-access-7kjl9\") pod \"cc1be430-a931-4237-937e-2597f724d57d\" (UID: \"cc1be430-a931-4237-937e-2597f724d57d\") " Nov 02 21:15:56.462344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.462313 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1be430-a931-4237-937e-2597f724d57d-maas-api-openshift-ca" pod "cc1be430-a931-4237-937e-2597f724d57d" (UID: "cc1be430-a931-4237-937e-2597f724d57d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:15:56.464078 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.464039 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1be430-a931-4237-937e-2597f724d57d-kube-api-access-7kjl9" pod "cc1be430-a931-4237-937e-2597f724d57d" (UID: "cc1be430-a931-4237-937e-2597f724d57d"). InnerVolumeSpecName "kube-api-access-7kjl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:15:56.464211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.464089 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1be430-a931-4237-937e-2597f724d57d-maas-api-xks-ca" pod "cc1be430-a931-4237-937e-2597f724d57d" (UID: "cc1be430-a931-4237-937e-2597f724d57d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:15:56.562779 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.562747 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1be430-a931-4237-937e-2597f724d57d-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:56.562779 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.562776 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1be430-a931-4237-937e-2597f724d57d-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:56.562779 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:56.562787 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjl9\" (UniqueName: \"kubernetes.io/projected/cc1be430-a931-4237-937e-2597f724d57d-kube-api-access-7kjl9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:15:57.188635 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:57.188606 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" Nov 02 21:15:57.188848 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:57.188637 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk" event={"ID":"cc1be430-a931-4237-937e-2597f724d57d","Type":"ContainerDied","Data":"0dd3f25b06af7fe679b6a06f029c870fb2f1c53445e970d3ec1cb820045a2892"} Nov 02 21:15:57.188848 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:15:57.188691 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd3f25b06af7fe679b6a06f029c870fb2f1c53445e970d3ec1cb820045a2892" Nov 02 21:16:03.883346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.883312 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd"] Nov 02 21:16:03.883801 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.883788 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cc1be430-a931-4237-937e-2597f724d57d" containerName="revoke-keys" Nov 02 21:16:03.883876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.883802 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1be430-a931-4237-937e-2597f724d57d" containerName="revoke-keys" Nov 02 21:16:03.883876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.883874 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1be430-a931-4237-937e-2597f724d57d" containerName="revoke-keys" Nov 02 21:16:03.887313 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.887295 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:03.936918 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.936875 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:16:03.937230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.936971 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:16:03.937600 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.937577 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:16:03.940417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.940393 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd"] Nov 02 21:16:03.983389 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:03.983354 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:16:04.037524 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.037473 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.037703 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.037638 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt7r8\" (UniqueName: \"kubernetes.io/projected/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-kube-api-access-lt7r8\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.037769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.037748 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.139106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.139017 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt7r8\" (UniqueName: \"kubernetes.io/projected/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-kube-api-access-lt7r8\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.139106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.139073 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.139506 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.139128 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.139960 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.139936 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.141757 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.141729 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.210340 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.210221 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt7r8\" (UniqueName: \"kubernetes.io/projected/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-kube-api-access-lt7r8\") pod \"maas-api-revoke-sub-e7afdf34-kgrfd\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.503690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.503659 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:04.880355 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:04.880321 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd"] Nov 02 21:16:04.882139 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:16:04.882105 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdc0d8e_685b_4ae0_ab76_709b2d3b79f4.slice/crio-509f304ac622b26034aa52ded8124a1e4b3b67eec47e4dea2a9e1430ed18d101 WatchSource:0}: Error finding container 509f304ac622b26034aa52ded8124a1e4b3b67eec47e4dea2a9e1430ed18d101: Status 404 returned error can't find the container with id 509f304ac622b26034aa52ded8124a1e4b3b67eec47e4dea2a9e1430ed18d101 Nov 02 21:16:05.246214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:05.246179 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" event={"ID":"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4","Type":"ContainerStarted","Data":"509f304ac622b26034aa52ded8124a1e4b3b67eec47e4dea2a9e1430ed18d101"} Nov 02 21:16:06.255388 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:06.255348 5800 generic.go:356] "Generic (PLEG): container finished" podID="efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" containerID="35528566236c050c88b82f8f5dd0289f1224596f2fef730fd1c06a691cf77dc9" exitCode=0 Nov 02 21:16:06.255856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:06.255431 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" event={"ID":"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4","Type":"ContainerDied","Data":"35528566236c050c88b82f8f5dd0289f1224596f2fef730fd1c06a691cf77dc9"} Nov 02 21:16:07.418169 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.418140 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:07.576482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.576399 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-xks-ca\") pod \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " Nov 02 21:16:07.576652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.576486 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-kube-api-access-lt7r8\" (UniqueName: \"kubernetes.io/projected/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-kube-api-access-lt7r8\") pod \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " Nov 02 21:16:07.576652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.576581 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-openshift-ca\") pod \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\" (UID: \"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4\") " Nov 02 21:16:07.576934 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.576904 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-openshift-ca" pod "efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" (UID: "efdc0d8e-685b-4ae0-ab76-709b2d3b79f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:16:07.578617 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.578581 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-kube-api-access-lt7r8" pod "efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" (UID: "efdc0d8e-685b-4ae0-ab76-709b2d3b79f4"). InnerVolumeSpecName "kube-api-access-lt7r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:16:07.578706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.578632 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-xks-ca" pod "efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" (UID: "efdc0d8e-685b-4ae0-ab76-709b2d3b79f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:16:07.677596 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.677559 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:07.677596 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.677591 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:07.677596 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:07.677601 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt7r8\" (UniqueName: \"kubernetes.io/projected/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4-kube-api-access-lt7r8\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:16:08.282016 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:08.281936 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" Nov 02 21:16:08.282016 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:08.281970 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd" event={"ID":"efdc0d8e-685b-4ae0-ab76-709b2d3b79f4","Type":"ContainerDied","Data":"509f304ac622b26034aa52ded8124a1e4b3b67eec47e4dea2a9e1430ed18d101"} Nov 02 21:16:08.282300 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:08.282030 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="509f304ac622b26034aa52ded8124a1e4b3b67eec47e4dea2a9e1430ed18d101" Nov 02 21:16:22.697271 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:16:22.697227 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:16:26.503680 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:26.503644 5800 scope.go:122] "RemoveContainer" containerID="08fc42ac3f965e0825d90983bbd3aa31706c2ade5b21ffa75359fbd34bcab5c8" Nov 02 21:16:26.513641 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:26.513617 5800 scope.go:122] "RemoveContainer" containerID="0225101bd754208b0224fbd5c0c866a093c4a730d6ea8a878f946b7e93d7a793" Nov 02 21:16:26.523036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:26.523010 5800 scope.go:122] "RemoveContainer" containerID="07ce150ca5eeccb1125bf3e8b35a501d483e0838f19e3b7e6a440a1717f2ef98" Nov 02 21:16:26.534856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:26.534831 5800 scope.go:122] "RemoveContainer" containerID="9dbbd998623f424d8b298987a28d9f687b064fb29c4290b171609d8ab8c3e58f" Nov 02 21:16:26.543909 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:26.543884 5800 scope.go:122] "RemoveContainer" containerID="7f6fbe3e0f82dae2310603b24bf78ffe96cb2d8b220d33cf01d40d047a05ea21" Nov 02 21:16:50.043448 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:50.043409 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9"] Nov 02 21:16:50.089790 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:50.089753 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ac3a97-78dp9"] Nov 02 21:16:51.622087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:16:51.622046 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45ffb9b-117e-48f4-9cba-1e22d49b31b3" path="/var/lib/kubelet/pods/d45ffb9b-117e-48f4-9cba-1e22d49b31b3/volumes" Nov 02 21:16:52.695535 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:16:52.695498 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:17:01.043740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:01.043665 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt"] Nov 02 21:17:01.082131 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:01.082092 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0179b70-pzgrt"] Nov 02 21:17:01.621934 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:01.621895 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5" path="/var/lib/kubelet/pods/a6e4ed1b-8db9-44e7-804a-468c8a9c0ee5/volumes" Nov 02 21:17:04.107356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:04.107315 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7"] Nov 02 21:17:09.107959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:09.107911 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" podUID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" containerName="payload-pre-processing" containerID="cri-o://e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3" gracePeriod=25 Nov 02 21:17:14.116482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:14.116430 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" podUID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:22.696850 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:17:22.696804 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:17:24.116653 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:24.116611 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" podUID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:24.971413 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:24.971370 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn"] Nov 02 21:17:24.971618 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:24.971585 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" containerID="cri-o://9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c" gracePeriod=30 Nov 02 21:17:26.592619 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:26.592579 5800 scope.go:122] "RemoveContainer" containerID="010f3aba3e96454c2a5e042b8c7f4713b434f1b1273a9972e4d4421706ddcab2" Nov 02 21:17:26.602841 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:26.602813 5800 scope.go:122] "RemoveContainer" containerID="2cd9169a029ca60b6e4555dfdbe5b0d8caf9e9a1f4b5ee4c5a39a3cba3a165a9" Nov 02 21:17:26.612453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:26.612429 5800 scope.go:122] "RemoveContainer" containerID="f65b356810304983d8395034fde1da2f9378a11dfa269fc75d756709ed579650" Nov 02 21:17:26.621149 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:26.621126 5800 scope.go:122] "RemoveContainer" containerID="76f70424e73638034d0818e6e9cdfac9cb6ad8c65b9707e1ed3d43af88ce82a1" Nov 02 21:17:26.629739 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:26.629715 5800 scope.go:122] "RemoveContainer" containerID="4c81d04e11e93e515c1639e0458161230c6240982759ddec3232dbbc98362bb5" Nov 02 21:17:29.383842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.383809 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 21:17:29.508268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.508232 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4ac609-e8d7-4d97-b0db-7963f40d7416-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4ac609-e8d7-4d97-b0db-7963f40d7416-plugins-config-volume\") pod \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " Nov 02 21:17:29.508478 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.508374 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4ac609-e8d7-4d97-b0db-7963f40d7416-kube-api-access-s9sfp\" (UniqueName: \"kubernetes.io/projected/ef4ac609-e8d7-4d97-b0db-7963f40d7416-kube-api-access-s9sfp\") pod \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\" (UID: \"ef4ac609-e8d7-4d97-b0db-7963f40d7416\") " Nov 02 21:17:29.508671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.508645 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4ac609-e8d7-4d97-b0db-7963f40d7416-plugins-config-volume" pod "ef4ac609-e8d7-4d97-b0db-7963f40d7416" (UID: "ef4ac609-e8d7-4d97-b0db-7963f40d7416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:29.510583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.510560 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4ac609-e8d7-4d97-b0db-7963f40d7416-kube-api-access-s9sfp" pod "ef4ac609-e8d7-4d97-b0db-7963f40d7416" (UID: "ef4ac609-e8d7-4d97-b0db-7963f40d7416"). InnerVolumeSpecName "kube-api-access-s9sfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:29.609855 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.609814 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef4ac609-e8d7-4d97-b0db-7963f40d7416-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:29.609855 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.609845 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9sfp\" (UniqueName: \"kubernetes.io/projected/ef4ac609-e8d7-4d97-b0db-7963f40d7416-kube-api-access-s9sfp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:29.817345 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.817306 5800 generic.go:356] "Generic (PLEG): container finished" podID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" containerID="e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3" exitCode=0 Nov 02 21:17:29.817521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.817355 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" Nov 02 21:17:29.817521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.817368 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" event={"ID":"ef4ac609-e8d7-4d97-b0db-7963f40d7416","Type":"ContainerDied","Data":"e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3"} Nov 02 21:17:29.817521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.817427 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7" event={"ID":"ef4ac609-e8d7-4d97-b0db-7963f40d7416","Type":"ContainerDied","Data":"685eb32857b08304226dd0bd22149127b61e6dbeb28dc644b0d3d01bf8762c86"} Nov 02 21:17:29.817521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.817452 5800 scope.go:122] "RemoveContainer" containerID="e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3" Nov 02 21:17:29.827303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.827264 5800 scope.go:122] "RemoveContainer" containerID="e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3" Nov 02 21:17:29.827591 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:17:29.827570 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3\": container with ID starting with e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3 not found: ID does not exist" containerID="e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3" Nov 02 21:17:29.827665 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.827603 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3"} err="failed to get container status \"e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3\": rpc error: code = NotFound desc = could not find container \"e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3\": container with ID starting with e6e065cebb983260765d1d78cbb4d974da31bf6bef4860e014f04880ab8e2ee3 not found: ID does not exist" Nov 02 21:17:29.970452 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:29.970369 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7"] Nov 02 21:17:30.001129 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.001099 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-76c687-6865f67c95-467d7"] Nov 02 21:17:30.235366 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.235309 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 21:17:30.316712 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.316675 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-credential-socket\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.316921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.316754 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-kube-api-access-xt9gh\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-kube-api-access-xt9gh\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.316921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.316788 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-socket\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.316921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.316811 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-envoy\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.316921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.316865 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-podinfo\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.316921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.316899 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2b029d-51c6-4959-8c8a-a8484a34907a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2b029d-51c6-4959-8c8a-a8484a34907a-istiod-ca-cert\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.317214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.316933 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-token\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-token\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.317214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317027 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-data\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.317214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317046 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-credential-socket" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:30.317214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317060 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-certs\") pod \"5c2b029d-51c6-4959-8c8a-a8484a34907a\" (UID: \"5c2b029d-51c6-4959-8c8a-a8484a34907a\") " Nov 02 21:17:30.317455 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317334 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-socket" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:30.317582 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317562 5800 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-credential-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.317662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317588 5800 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-socket\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.317662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317568 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2b029d-51c6-4959-8c8a-a8484a34907a-istiod-ca-cert" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:30.317816 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317791 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-certs" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:30.317993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.317970 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-data" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:30.319720 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.319691 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-podinfo" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:17:30.319839 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.319716 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-envoy" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:17:30.320069 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.320049 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-token" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:30.320158 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.320079 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-kube-api-access-xt9gh" pod "5c2b029d-51c6-4959-8c8a-a8484a34907a" (UID: "5c2b029d-51c6-4959-8c8a-a8484a34907a"). InnerVolumeSpecName "kube-api-access-xt9gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:30.418729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.418689 5800 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-podinfo\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.418729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.418723 5800 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2b029d-51c6-4959-8c8a-a8484a34907a-istiod-ca-cert\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.418729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.418732 5800 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-token\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.418729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.418740 5800 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-data\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.419235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.418749 5800 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-workload-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.419235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.418758 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt9gh\" (UniqueName: \"kubernetes.io/projected/5c2b029d-51c6-4959-8c8a-a8484a34907a-kube-api-access-xt9gh\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.419235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.418769 5800 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2b029d-51c6-4959-8c8a-a8484a34907a-istio-envoy\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:30.825595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.825554 5800 generic.go:356] "Generic (PLEG): container finished" podID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerID="9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c" exitCode=0 Nov 02 21:17:30.825798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.825617 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" Nov 02 21:17:30.825798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.825635 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" event={"ID":"5c2b029d-51c6-4959-8c8a-a8484a34907a","Type":"ContainerDied","Data":"9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c"} Nov 02 21:17:30.825798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.825696 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn" event={"ID":"5c2b029d-51c6-4959-8c8a-a8484a34907a","Type":"ContainerDied","Data":"3cefc4c22f0ba50345b1b2c9f0a15f4819cd5adb2b28579e76a942c4451ef598"} Nov 02 21:17:30.825798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.825729 5800 scope.go:122] "RemoveContainer" containerID="9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c" Nov 02 21:17:30.837042 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.837018 5800 scope.go:122] "RemoveContainer" containerID="f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6" Nov 02 21:17:30.849545 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.849517 5800 scope.go:122] "RemoveContainer" containerID="9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c" Nov 02 21:17:30.849886 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:17:30.849859 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c\": container with ID starting with 9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c not found: ID does not exist" containerID="9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c" Nov 02 21:17:30.849957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.849900 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c"} err="failed to get container status \"9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c\": rpc error: code = NotFound desc = could not find container \"9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c\": container with ID starting with 9f7760c5c753f69abc6c6ff486b39130ebbd2ea82bc6be7fdbdde8bac43b086c not found: ID does not exist" Nov 02 21:17:30.849957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.849926 5800 scope.go:122] "RemoveContainer" containerID="f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6" Nov 02 21:17:30.850618 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:17:30.850590 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6\": container with ID starting with f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6 not found: ID does not exist" containerID="f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6" Nov 02 21:17:30.850730 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.850628 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6"} err="failed to get container status \"f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6\": rpc error: code = NotFound desc = could not find container \"f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6\": container with ID starting with f24d88efc36b70e19b2771ff9a6e06e7db0d292603e1f579ae1a312a7318d0c6 not found: ID does not exist" Nov 02 21:17:30.981102 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:30.981061 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn"] Nov 02 21:17:31.027065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:31.027026 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-76c687-openshift-default-564cd798d9-kknkn"] Nov 02 21:17:31.621113 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:31.621078 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" path="/var/lib/kubelet/pods/5c2b029d-51c6-4959-8c8a-a8484a34907a/volumes" Nov 02 21:17:31.621556 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:31.621540 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" path="/var/lib/kubelet/pods/ef4ac609-e8d7-4d97-b0db-7963f40d7416/volumes" Nov 02 21:17:32.360437 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:32.360397 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts"] Nov 02 21:17:37.361141 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:37.361093 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" podUID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" containerName="payload-pre-processing" containerID="cri-o://1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb" gracePeriod=25 Nov 02 21:17:39.591383 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:39.591345 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" podUID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:49.591933 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:49.591898 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" podUID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:17:52.695925 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:17:52.695888 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:17:54.374091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374039 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw"] Nov 02 21:17:54.374647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374510 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" containerName="payload-pre-processing" Nov 02 21:17:54.374647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374526 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" containerName="payload-pre-processing" Nov 02 21:17:54.374647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374545 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" Nov 02 21:17:54.374647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374551 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" Nov 02 21:17:54.374647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374582 5800 cpu_manager.go:414] "containerMap: removing container" podUID="efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" containerName="revoke-keys" Nov 02 21:17:54.374647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374587 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" containerName="revoke-keys" Nov 02 21:17:54.374992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374659 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" Nov 02 21:17:54.374992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374667 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" containerName="revoke-keys" Nov 02 21:17:54.374992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374677 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4ac609-e8d7-4d97-b0db-7963f40d7416" containerName="payload-pre-processing" Nov 02 21:17:54.374992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.374687 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" Nov 02 21:17:54.378051 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.378028 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.430898 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.430854 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw"] Nov 02 21:17:54.438795 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.438765 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhn8\" (UniqueName: \"kubernetes.io/projected/5796aa16-b57e-4619-ae6f-97f820af19be-kube-api-access-cfhn8\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.438976 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.438837 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.438976 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.438903 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.473575 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.473546 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:17:54.473764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.473596 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:17:54.473764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.473626 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:17:54.473764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.473643 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:17:54.540140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.540106 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.540365 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.540161 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhn8\" (UniqueName: \"kubernetes.io/projected/5796aa16-b57e-4619-ae6f-97f820af19be-kube-api-access-cfhn8\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.540365 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.540203 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.540781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.540759 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.542511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.542488 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.593498 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.593466 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhn8\" (UniqueName: \"kubernetes.io/projected/5796aa16-b57e-4619-ae6f-97f820af19be-kube-api-access-cfhn8\") pod \"maas-api-revoke-sub-f67db470-88vxw\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:54.691479 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:54.691401 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:55.055377 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:55.055338 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw"] Nov 02 21:17:55.057644 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:17:55.057616 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5796aa16_b57e_4619_ae6f_97f820af19be.slice/crio-a4bf0748341c0eca7d440dc8f26609ef5b7f7ba37a50b3c77f7de7615b1b9456 WatchSource:0}: Error finding container a4bf0748341c0eca7d440dc8f26609ef5b7f7ba37a50b3c77f7de7615b1b9456: Status 404 returned error can't find the container with id a4bf0748341c0eca7d440dc8f26609ef5b7f7ba37a50b3c77f7de7615b1b9456 Nov 02 21:17:55.975015 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:55.974974 5800 generic.go:356] "Generic (PLEG): container finished" podID="5796aa16-b57e-4619-ae6f-97f820af19be" containerID="3493ee1d0b1eabaf2e2df53fdafe98667c5d99d3b901b97823efea0da0a74c71" exitCode=0 Nov 02 21:17:55.975400 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:55.975051 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" event={"ID":"5796aa16-b57e-4619-ae6f-97f820af19be","Type":"ContainerDied","Data":"3493ee1d0b1eabaf2e2df53fdafe98667c5d99d3b901b97823efea0da0a74c71"} Nov 02 21:17:55.975400 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:55.975103 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" event={"ID":"5796aa16-b57e-4619-ae6f-97f820af19be","Type":"ContainerStarted","Data":"a4bf0748341c0eca7d440dc8f26609ef5b7f7ba37a50b3c77f7de7615b1b9456"} Nov 02 21:17:57.131545 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.131519 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:57.167710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.167675 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-openshift-ca\") pod \"5796aa16-b57e-4619-ae6f-97f820af19be\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " Nov 02 21:17:57.167873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.167808 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5796aa16-b57e-4619-ae6f-97f820af19be-kube-api-access-cfhn8\" (UniqueName: \"kubernetes.io/projected/5796aa16-b57e-4619-ae6f-97f820af19be-kube-api-access-cfhn8\") pod \"5796aa16-b57e-4619-ae6f-97f820af19be\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " Nov 02 21:17:57.167873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.167836 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-xks-ca\") pod \"5796aa16-b57e-4619-ae6f-97f820af19be\" (UID: \"5796aa16-b57e-4619-ae6f-97f820af19be\") " Nov 02 21:17:57.168130 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.168101 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-openshift-ca" pod "5796aa16-b57e-4619-ae6f-97f820af19be" (UID: "5796aa16-b57e-4619-ae6f-97f820af19be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:57.169931 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.169905 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5796aa16-b57e-4619-ae6f-97f820af19be-kube-api-access-cfhn8" pod "5796aa16-b57e-4619-ae6f-97f820af19be" (UID: "5796aa16-b57e-4619-ae6f-97f820af19be"). InnerVolumeSpecName "kube-api-access-cfhn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:57.170058 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.169908 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-xks-ca" pod "5796aa16-b57e-4619-ae6f-97f820af19be" (UID: "5796aa16-b57e-4619-ae6f-97f820af19be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:17:57.269408 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.269371 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:57.269408 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.269403 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhn8\" (UniqueName: \"kubernetes.io/projected/5796aa16-b57e-4619-ae6f-97f820af19be-kube-api-access-cfhn8\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:57.269408 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.269414 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5796aa16-b57e-4619-ae6f-97f820af19be-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:57.609269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.609237 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 21:17:57.674129 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.674091 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31669d6-c91d-4a6b-9a10-282e9bb46d76-kube-api-access-zcjjm\" (UniqueName: \"kubernetes.io/projected/c31669d6-c91d-4a6b-9a10-282e9bb46d76-kube-api-access-zcjjm\") pod \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " Nov 02 21:17:57.674332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.674144 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31669d6-c91d-4a6b-9a10-282e9bb46d76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31669d6-c91d-4a6b-9a10-282e9bb46d76-plugins-config-volume\") pod \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\" (UID: \"c31669d6-c91d-4a6b-9a10-282e9bb46d76\") " Nov 02 21:17:57.674605 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.674571 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31669d6-c91d-4a6b-9a10-282e9bb46d76-plugins-config-volume" pod "c31669d6-c91d-4a6b-9a10-282e9bb46d76" (UID: "c31669d6-c91d-4a6b-9a10-282e9bb46d76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:17:57.676940 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.676906 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31669d6-c91d-4a6b-9a10-282e9bb46d76-kube-api-access-zcjjm" pod "c31669d6-c91d-4a6b-9a10-282e9bb46d76" (UID: "c31669d6-c91d-4a6b-9a10-282e9bb46d76"). InnerVolumeSpecName "kube-api-access-zcjjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:17:57.775663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.775625 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcjjm\" (UniqueName: \"kubernetes.io/projected/c31669d6-c91d-4a6b-9a10-282e9bb46d76-kube-api-access-zcjjm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:57.775663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.775658 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31669d6-c91d-4a6b-9a10-282e9bb46d76-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:17:57.990003 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.989906 5800 generic.go:356] "Generic (PLEG): container finished" podID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" containerID="1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb" exitCode=0 Nov 02 21:17:57.990003 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.989976 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" Nov 02 21:17:57.990003 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.989995 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" event={"ID":"c31669d6-c91d-4a6b-9a10-282e9bb46d76","Type":"ContainerDied","Data":"1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb"} Nov 02 21:17:57.990339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.990082 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts" event={"ID":"c31669d6-c91d-4a6b-9a10-282e9bb46d76","Type":"ContainerDied","Data":"d1c865ac92117723e31f689ae11f7b013ec8f73165ca376015e251f0d45bd168"} Nov 02 21:17:57.990339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.990110 5800 scope.go:122] "RemoveContainer" containerID="1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb" Nov 02 21:17:57.991601 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.991452 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" Nov 02 21:17:57.991601 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.991492 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw" event={"ID":"5796aa16-b57e-4619-ae6f-97f820af19be","Type":"ContainerDied","Data":"a4bf0748341c0eca7d440dc8f26609ef5b7f7ba37a50b3c77f7de7615b1b9456"} Nov 02 21:17:57.991601 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:57.991541 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4bf0748341c0eca7d440dc8f26609ef5b7f7ba37a50b3c77f7de7615b1b9456" Nov 02 21:17:58.000443 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:58.000422 5800 scope.go:122] "RemoveContainer" containerID="1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb" Nov 02 21:17:58.000742 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:17:58.000723 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb\": container with ID starting with 1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb not found: ID does not exist" containerID="1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb" Nov 02 21:17:58.000802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:58.000750 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb"} err="failed to get container status \"1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb\": rpc error: code = NotFound desc = could not find container \"1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb\": container with ID starting with 1c2fab1f27f1885899eccb5686d5dd350ab6d78c14894b562f60210ff397fbcb not found: ID does not exist" Nov 02 21:17:58.143464 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:58.143424 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts"] Nov 02 21:17:58.189736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:58.189705 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-6b12ae-5c796465b7-jxxts"] Nov 02 21:17:58.679736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:58.679700 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc"] Nov 02 21:17:59.622323 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:17:59.622271 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" path="/var/lib/kubelet/pods/c31669d6-c91d-4a6b-9a10-282e9bb46d76/volumes" Nov 02 21:18:06.322627 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.322593 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn"] Nov 02 21:18:06.323050 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.322999 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5796aa16-b57e-4619-ae6f-97f820af19be" containerName="revoke-keys" Nov 02 21:18:06.323050 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.323009 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5796aa16-b57e-4619-ae6f-97f820af19be" containerName="revoke-keys" Nov 02 21:18:06.323050 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.323022 5800 cpu_manager.go:414] "containerMap: removing container" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" Nov 02 21:18:06.323050 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.323027 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2b029d-51c6-4959-8c8a-a8484a34907a" containerName="istio-proxy" Nov 02 21:18:06.323050 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.323049 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" containerName="payload-pre-processing" Nov 02 21:18:06.323226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.323055 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" containerName="payload-pre-processing" Nov 02 21:18:06.323226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.323136 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="5796aa16-b57e-4619-ae6f-97f820af19be" containerName="revoke-keys" Nov 02 21:18:06.323226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.323150 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31669d6-c91d-4a6b-9a10-282e9bb46d76" containerName="payload-pre-processing" Nov 02 21:18:06.326336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.326317 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.423104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.423072 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:18:06.423104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.423080 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:18:06.423401 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.423266 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:18:06.423401 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.423276 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nnvfd\"" Nov 02 21:18:06.425822 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.425798 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn"] Nov 02 21:18:06.463113 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.463082 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.463263 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.463116 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ds2\" (UniqueName: \"kubernetes.io/projected/2d0ddf21-c52f-49fe-89af-482bbddfc0be-kube-api-access-74ds2\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.463263 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.463139 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.564532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.564470 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.564721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.564549 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ds2\" (UniqueName: \"kubernetes.io/projected/2d0ddf21-c52f-49fe-89af-482bbddfc0be-kube-api-access-74ds2\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.564721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.564582 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.565166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.565143 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.567036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.567017 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.617573 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.617504 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ds2\" (UniqueName: \"kubernetes.io/projected/2d0ddf21-c52f-49fe-89af-482bbddfc0be-kube-api-access-74ds2\") pod \"maas-api-revoke-sub-605538c6-wxgwn\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:06.639124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:06.639078 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:07.006810 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:07.006781 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn"] Nov 02 21:18:07.009504 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:18:07.009473 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0ddf21_c52f_49fe_89af_482bbddfc0be.slice/crio-b2adecc05a0f050bed1dfc71ba804b8a070b50be51d09f5df7c42c3940c878ba WatchSource:0}: Error finding container b2adecc05a0f050bed1dfc71ba804b8a070b50be51d09f5df7c42c3940c878ba: Status 404 returned error can't find the container with id b2adecc05a0f050bed1dfc71ba804b8a070b50be51d09f5df7c42c3940c878ba Nov 02 21:18:07.046094 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:07.046055 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" event={"ID":"2d0ddf21-c52f-49fe-89af-482bbddfc0be","Type":"ContainerStarted","Data":"b2adecc05a0f050bed1dfc71ba804b8a070b50be51d09f5df7c42c3940c878ba"} Nov 02 21:18:07.579361 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:07.579328 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv"] Nov 02 21:18:07.678239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:07.678207 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv"] Nov 02 21:18:08.054356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:08.054317 5800 generic.go:356] "Generic (PLEG): container finished" podID="2d0ddf21-c52f-49fe-89af-482bbddfc0be" containerID="30a6c2e9ab36b70cf0dd8133cbde9a4bc74435001694c6d4c580ec9daf34220f" exitCode=0 Nov 02 21:18:08.054533 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:08.054402 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" event={"ID":"2d0ddf21-c52f-49fe-89af-482bbddfc0be","Type":"ContainerDied","Data":"30a6c2e9ab36b70cf0dd8133cbde9a4bc74435001694c6d4c580ec9daf34220f"} Nov 02 21:18:09.209561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.209528 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:09.293535 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.293503 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-openshift-ca\") pod \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " Nov 02 21:18:09.293740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.293588 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0ddf21-c52f-49fe-89af-482bbddfc0be-kube-api-access-74ds2\" (UniqueName: \"kubernetes.io/projected/2d0ddf21-c52f-49fe-89af-482bbddfc0be-kube-api-access-74ds2\") pod \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " Nov 02 21:18:09.293740 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.293671 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-xks-ca\") pod \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\" (UID: \"2d0ddf21-c52f-49fe-89af-482bbddfc0be\") " Nov 02 21:18:09.293939 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.293906 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-openshift-ca" pod "2d0ddf21-c52f-49fe-89af-482bbddfc0be" (UID: "2d0ddf21-c52f-49fe-89af-482bbddfc0be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:18:09.295734 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.295709 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0ddf21-c52f-49fe-89af-482bbddfc0be-kube-api-access-74ds2" pod "2d0ddf21-c52f-49fe-89af-482bbddfc0be" (UID: "2d0ddf21-c52f-49fe-89af-482bbddfc0be"). InnerVolumeSpecName "kube-api-access-74ds2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:09.295905 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.295876 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-xks-ca" pod "2d0ddf21-c52f-49fe-89af-482bbddfc0be" (UID: "2d0ddf21-c52f-49fe-89af-482bbddfc0be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:18:09.395356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.395230 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:09.395356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.395263 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d0ddf21-c52f-49fe-89af-482bbddfc0be-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:09.395356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:09.395275 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ds2\" (UniqueName: \"kubernetes.io/projected/2d0ddf21-c52f-49fe-89af-482bbddfc0be-kube-api-access-74ds2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:10.069156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:10.069123 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" Nov 02 21:18:10.069414 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:10.069152 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn" event={"ID":"2d0ddf21-c52f-49fe-89af-482bbddfc0be","Type":"ContainerDied","Data":"b2adecc05a0f050bed1dfc71ba804b8a070b50be51d09f5df7c42c3940c878ba"} Nov 02 21:18:10.069414 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:10.069201 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2adecc05a0f050bed1dfc71ba804b8a070b50be51d09f5df7c42c3940c878ba" Nov 02 21:18:12.579816 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:12.579743 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" podUID="662613f8-7642-4d75-86b9-f6fe2637d130" containerName="payload-processing" containerID="cri-o://735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4" gracePeriod=25 Nov 02 21:18:12.679226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:12.679180 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" podUID="763ecd5b-ac98-4c72-8271-07b64797d0bf" containerName="payload-pre-processing" containerID="cri-o://f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1" gracePeriod=25 Nov 02 21:18:13.700064 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:13.700018 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" containerID="cri-o://dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e" gracePeriod=45 Nov 02 21:18:13.969728 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:13.969698 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 21:18:14.044426 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.044356 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-home\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-home\") pod \"ac5af302-63fa-46da-883a-740f4c486be5\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " Nov 02 21:18:14.044696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.044437 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5af302-63fa-46da-883a-740f4c486be5-kube-api-access-sq74r\" (UniqueName: \"kubernetes.io/projected/ac5af302-63fa-46da-883a-740f4c486be5-kube-api-access-sq74r\") pod \"ac5af302-63fa-46da-883a-740f4c486be5\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " Nov 02 21:18:14.044696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.044482 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5af302-63fa-46da-883a-740f4c486be5-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5af302-63fa-46da-883a-740f4c486be5-tls-certs\") pod \"ac5af302-63fa-46da-883a-740f4c486be5\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " Nov 02 21:18:14.044696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.044557 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-tmp-dir\") pod \"ac5af302-63fa-46da-883a-740f4c486be5\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " Nov 02 21:18:14.044696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.044627 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-model-cache\") pod \"ac5af302-63fa-46da-883a-740f4c486be5\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " Nov 02 21:18:14.044696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.044642 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-home" pod "ac5af302-63fa-46da-883a-740f4c486be5" (UID: "ac5af302-63fa-46da-883a-740f4c486be5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:14.044696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.044682 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-dshm\") pod \"ac5af302-63fa-46da-883a-740f4c486be5\" (UID: \"ac5af302-63fa-46da-883a-740f4c486be5\") " Nov 02 21:18:14.046299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.045142 5800 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-home\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.046299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.045216 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-tmp-dir" pod "ac5af302-63fa-46da-883a-740f4c486be5" (UID: "ac5af302-63fa-46da-883a-740f4c486be5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:14.046299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.045975 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-model-cache" pod "ac5af302-63fa-46da-883a-740f4c486be5" (UID: "ac5af302-63fa-46da-883a-740f4c486be5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:14.048209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.048184 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-dshm" pod "ac5af302-63fa-46da-883a-740f4c486be5" (UID: "ac5af302-63fa-46da-883a-740f4c486be5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:18:14.048510 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.048460 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5af302-63fa-46da-883a-740f4c486be5-kube-api-access-sq74r" pod "ac5af302-63fa-46da-883a-740f4c486be5" (UID: "ac5af302-63fa-46da-883a-740f4c486be5"). InnerVolumeSpecName "kube-api-access-sq74r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:14.049915 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.049077 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5af302-63fa-46da-883a-740f4c486be5-tls-certs" pod "ac5af302-63fa-46da-883a-740f4c486be5" (UID: "ac5af302-63fa-46da-883a-740f4c486be5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:18:14.096764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.096723 5800 generic.go:356] "Generic (PLEG): container finished" podID="ac5af302-63fa-46da-883a-740f4c486be5" containerID="dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e" exitCode=0 Nov 02 21:18:14.096963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.096772 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" Nov 02 21:18:14.096963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.096815 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" event={"ID":"ac5af302-63fa-46da-883a-740f4c486be5","Type":"ContainerDied","Data":"dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e"} Nov 02 21:18:14.096963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.096872 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc" event={"ID":"ac5af302-63fa-46da-883a-740f4c486be5","Type":"ContainerDied","Data":"74d0ec3680ebf0d5568bb837bc34748e3550453edc3ec08931041a846158e0fd"} Nov 02 21:18:14.096963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.096896 5800 scope.go:122] "RemoveContainer" containerID="dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e" Nov 02 21:18:14.109242 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.109214 5800 scope.go:122] "RemoveContainer" containerID="dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e" Nov 02 21:18:14.109607 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:18:14.109582 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e\": container with ID starting with dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e not found: ID does not exist" containerID="dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e" Nov 02 21:18:14.109681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.109619 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e"} err="failed to get container status \"dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e\": rpc error: code = NotFound desc = could not find container \"dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e\": container with ID starting with dabdd171525777fbc90c047d69282fa6439259a30fb8cb7640ca88cf32d1873e not found: ID does not exist" Nov 02 21:18:14.146024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.145885 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq74r\" (UniqueName: \"kubernetes.io/projected/ac5af302-63fa-46da-883a-740f4c486be5-kube-api-access-sq74r\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.146024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.145926 5800 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5af302-63fa-46da-883a-740f4c486be5-tls-certs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.146024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.145966 5800 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-tmp-dir\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.146024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.145983 5800 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-model-cache\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.146024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.145997 5800 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5af302-63fa-46da-883a-740f4c486be5-dshm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:14.249986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.249951 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc"] Nov 02 21:18:14.259771 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:14.259731 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-14d090/premium-w0-14d090-kserve-59869d8966-2dmrc"] Nov 02 21:18:15.621384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:15.621348 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5af302-63fa-46da-883a-740f4c486be5" path="/var/lib/kubelet/pods/ac5af302-63fa-46da-883a-740f4c486be5/volumes" Nov 02 21:18:17.057051 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:17.057015 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj"] Nov 02 21:18:17.103532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:17.103496 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-562a6a3f-ptncj"] Nov 02 21:18:17.620632 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:17.620597 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2189fa77-2f5d-49eb-bafc-b30194ee9d41" path="/var/lib/kubelet/pods/2189fa77-2f5d-49eb-bafc-b30194ee9d41/volumes" Nov 02 21:18:18.796060 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:18.796016 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" podUID="662613f8-7642-4d75-86b9-f6fe2637d130" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:18:18.796511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:18.796027 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" podUID="763ecd5b-ac98-4c72-8271-07b64797d0bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:18:22.648103 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:22.648066 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:18:22.654736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:22.654709 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:18:22.695826 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:18:22.695789 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:18:26.672481 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:26.672451 5800 scope.go:122] "RemoveContainer" containerID="441f4efacba9dd0f4a28a4c996f2230e4cab48db1dc6ae504dfee1aaecebad96" Nov 02 21:18:26.682320 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:26.682299 5800 scope.go:122] "RemoveContainer" containerID="e108a4fdd9444e59d1efb102d8d7c320836c838578b442de1a4009c1b2152812" Nov 02 21:18:27.273766 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:27.273731 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f74f7c786-fr9rr_44b1f0d7-2146-4520-bfc1-f7c56ddd37be/authorino/0.log" Nov 02 21:18:27.639289 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:27.639195 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2mk96_cbda65e8-653a-4cf7-8e99-fecf2db77fd9/manager/0.log" Nov 02 21:18:28.795539 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:28.795494 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" podUID="763ecd5b-ac98-4c72-8271-07b64797d0bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:18:28.795539 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:28.795522 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" podUID="662613f8-7642-4d75-86b9-f6fe2637d130" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:18:29.076141 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:29.076063 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-vnq4k_19a2f253-46f8-4c5f-ad82-dcd0bac32f1b/manager/0.log" Nov 02 21:18:32.828621 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:32.828578 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 21:18:32.929787 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:32.929755 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662613f8-7642-4d75-86b9-f6fe2637d130-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662613f8-7642-4d75-86b9-f6fe2637d130-plugins-config-volume\") pod \"662613f8-7642-4d75-86b9-f6fe2637d130\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " Nov 02 21:18:32.929935 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:32.929861 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662613f8-7642-4d75-86b9-f6fe2637d130-kube-api-access-t4m6h\" (UniqueName: \"kubernetes.io/projected/662613f8-7642-4d75-86b9-f6fe2637d130-kube-api-access-t4m6h\") pod \"662613f8-7642-4d75-86b9-f6fe2637d130\" (UID: \"662613f8-7642-4d75-86b9-f6fe2637d130\") " Nov 02 21:18:32.930205 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:32.930179 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662613f8-7642-4d75-86b9-f6fe2637d130-plugins-config-volume" pod "662613f8-7642-4d75-86b9-f6fe2637d130" (UID: "662613f8-7642-4d75-86b9-f6fe2637d130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:18:32.931980 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:32.931957 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662613f8-7642-4d75-86b9-f6fe2637d130-kube-api-access-t4m6h" pod "662613f8-7642-4d75-86b9-f6fe2637d130" (UID: "662613f8-7642-4d75-86b9-f6fe2637d130"). InnerVolumeSpecName "kube-api-access-t4m6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:32.932600 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:32.932584 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 21:18:33.030887 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.030853 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763ecd5b-ac98-4c72-8271-07b64797d0bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ecd5b-ac98-4c72-8271-07b64797d0bf-plugins-config-volume\") pod \"763ecd5b-ac98-4c72-8271-07b64797d0bf\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " Nov 02 21:18:33.031072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.031056 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763ecd5b-ac98-4c72-8271-07b64797d0bf-kube-api-access-6pzr4\" (UniqueName: \"kubernetes.io/projected/763ecd5b-ac98-4c72-8271-07b64797d0bf-kube-api-access-6pzr4\") pod \"763ecd5b-ac98-4c72-8271-07b64797d0bf\" (UID: \"763ecd5b-ac98-4c72-8271-07b64797d0bf\") " Nov 02 21:18:33.031339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.031317 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763ecd5b-ac98-4c72-8271-07b64797d0bf-plugins-config-volume" pod "763ecd5b-ac98-4c72-8271-07b64797d0bf" (UID: "763ecd5b-ac98-4c72-8271-07b64797d0bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:18:33.031398 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.031384 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4m6h\" (UniqueName: \"kubernetes.io/projected/662613f8-7642-4d75-86b9-f6fe2637d130-kube-api-access-t4m6h\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:33.031444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.031399 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763ecd5b-ac98-4c72-8271-07b64797d0bf-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:33.031444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.031415 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662613f8-7642-4d75-86b9-f6fe2637d130-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:33.033124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.033101 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763ecd5b-ac98-4c72-8271-07b64797d0bf-kube-api-access-6pzr4" pod "763ecd5b-ac98-4c72-8271-07b64797d0bf" (UID: "763ecd5b-ac98-4c72-8271-07b64797d0bf"). InnerVolumeSpecName "kube-api-access-6pzr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:18:33.047167 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.047137 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl"] Nov 02 21:18:33.058363 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.052889 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512e10c2-zfjnl"] Nov 02 21:18:33.132954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.132874 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzr4\" (UniqueName: \"kubernetes.io/projected/763ecd5b-ac98-4c72-8271-07b64797d0bf-kube-api-access-6pzr4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:18:33.212396 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.212360 5800 generic.go:356] "Generic (PLEG): container finished" podID="763ecd5b-ac98-4c72-8271-07b64797d0bf" containerID="f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1" exitCode=0 Nov 02 21:18:33.212606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.212434 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" Nov 02 21:18:33.212606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.212445 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" event={"ID":"763ecd5b-ac98-4c72-8271-07b64797d0bf","Type":"ContainerDied","Data":"f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1"} Nov 02 21:18:33.212606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.212534 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv" event={"ID":"763ecd5b-ac98-4c72-8271-07b64797d0bf","Type":"ContainerDied","Data":"a3f3026a37f2f90763d5d168849ed91ade9feb9ac931da84b6dc1c8affe7da14"} Nov 02 21:18:33.212606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.212569 5800 scope.go:122] "RemoveContainer" containerID="f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1" Nov 02 21:18:33.213864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.213843 5800 generic.go:356] "Generic (PLEG): container finished" podID="662613f8-7642-4d75-86b9-f6fe2637d130" containerID="735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4" exitCode=0 Nov 02 21:18:33.213957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.213921 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" Nov 02 21:18:33.214024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.213924 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" event={"ID":"662613f8-7642-4d75-86b9-f6fe2637d130","Type":"ContainerDied","Data":"735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4"} Nov 02 21:18:33.214098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.214034 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv" event={"ID":"662613f8-7642-4d75-86b9-f6fe2637d130","Type":"ContainerDied","Data":"fe4de189e7bc3d28ca10e11821e6e2e1bd2716ad4c487e133648b960877501cc"} Nov 02 21:18:33.223736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.223714 5800 scope.go:122] "RemoveContainer" containerID="f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1" Nov 02 21:18:33.224055 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:18:33.224011 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1\": container with ID starting with f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1 not found: ID does not exist" containerID="f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1" Nov 02 21:18:33.224191 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.224065 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1"} err="failed to get container status \"f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1\": rpc error: code = NotFound desc = could not find container \"f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1\": container with ID starting with f76a3c418918e8799c8f1d99bc8e349791532b2655da51c6c766faea0aa8c6a1 not found: ID does not exist" Nov 02 21:18:33.224191 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.224089 5800 scope.go:122] "RemoveContainer" containerID="735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4" Nov 02 21:18:33.232474 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.232457 5800 scope.go:122] "RemoveContainer" containerID="735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4" Nov 02 21:18:33.232704 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:18:33.232684 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4\": container with ID starting with 735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4 not found: ID does not exist" containerID="735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4" Nov 02 21:18:33.232759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.232709 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4"} err="failed to get container status \"735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4\": rpc error: code = NotFound desc = could not find container \"735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4\": container with ID starting with 735245f4c85d6202043b85d63cb1380aefa4f09162b82b25d7f9153095a8b8e4 not found: ID does not exist" Nov 02 21:18:33.365710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.365617 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv"] Nov 02 21:18:33.368766 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.368742 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-14d090-6f9f5b46f9-fp4cv"] Nov 02 21:18:33.469635 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.469532 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv"] Nov 02 21:18:33.473604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.473574 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-14d090-74cb777bc5-k8wgv"] Nov 02 21:18:33.620185 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.620155 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f832e7f-806d-47f0-b6bd-8203a42da733" path="/var/lib/kubelet/pods/0f832e7f-806d-47f0-b6bd-8203a42da733/volumes" Nov 02 21:18:33.620596 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.620582 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662613f8-7642-4d75-86b9-f6fe2637d130" path="/var/lib/kubelet/pods/662613f8-7642-4d75-86b9-f6fe2637d130/volumes" Nov 02 21:18:33.620940 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:18:33.620927 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763ecd5b-ac98-4c72-8271-07b64797d0bf" path="/var/lib/kubelet/pods/763ecd5b-ac98-4c72-8271-07b64797d0bf/volumes" Nov 02 21:18:52.695814 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:18:52.695771 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:19:17.721778 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.721743 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9"] Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722141 5800 cpu_manager.go:414] "containerMap: removing container" podUID="662613f8-7642-4d75-86b9-f6fe2637d130" containerName="payload-processing" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722153 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662613f8-7642-4d75-86b9-f6fe2637d130" containerName="payload-processing" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722178 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2d0ddf21-c52f-49fe-89af-482bbddfc0be" containerName="revoke-keys" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722184 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0ddf21-c52f-49fe-89af-482bbddfc0be" containerName="revoke-keys" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722193 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722198 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722206 5800 cpu_manager.go:414] "containerMap: removing container" podUID="763ecd5b-ac98-4c72-8271-07b64797d0bf" containerName="payload-pre-processing" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722211 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763ecd5b-ac98-4c72-8271-07b64797d0bf" containerName="payload-pre-processing" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722276 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5af302-63fa-46da-883a-740f4c486be5" containerName="main" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722296 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="763ecd5b-ac98-4c72-8271-07b64797d0bf" containerName="payload-pre-processing" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722307 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="662613f8-7642-4d75-86b9-f6fe2637d130" containerName="payload-processing" Nov 02 21:19:17.722335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.722317 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0ddf21-c52f-49fe-89af-482bbddfc0be" containerName="revoke-keys" Nov 02 21:19:17.725444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.725427 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.783067 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.783032 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbml\" (UniqueName: \"kubernetes.io/projected/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-kube-api-access-hnbml\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.783308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.783190 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.783308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.783229 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.824667 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.824630 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:19:17.824873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.824706 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:19:17.824873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.824725 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:19:17.824873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.824791 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:19:17.827403 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.827378 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9"] Nov 02 21:19:17.883831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.883788 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.883831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.883832 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.884062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.883885 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbml\" (UniqueName: \"kubernetes.io/projected/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-kube-api-access-hnbml\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.884572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.884542 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.886349 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.886318 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:17.937122 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:17.937081 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbml\" (UniqueName: \"kubernetes.io/projected/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-kube-api-access-hnbml\") pod \"maas-api-revoke-keys-e2e-ait-056961db-tr9p9\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:18.038396 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:18.038360 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:18.412406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:18.412372 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9"] Nov 02 21:19:18.414508 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:19:18.414477 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb61ea2f_e4f6_4204_b5e2_e047fa0efe02.slice/crio-b720da4481140ad72826909230ed0403c4db38514d2742b229bd4566b8267430 WatchSource:0}: Error finding container b720da4481140ad72826909230ed0403c4db38514d2742b229bd4566b8267430: Status 404 returned error can't find the container with id b720da4481140ad72826909230ed0403c4db38514d2742b229bd4566b8267430 Nov 02 21:19:18.477470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:18.477429 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" event={"ID":"db61ea2f-e4f6-4204-b5e2-e047fa0efe02","Type":"ContainerStarted","Data":"b720da4481140ad72826909230ed0403c4db38514d2742b229bd4566b8267430"} Nov 02 21:19:19.484640 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:19.484545 5800 generic.go:356] "Generic (PLEG): container finished" podID="db61ea2f-e4f6-4204-b5e2-e047fa0efe02" containerID="b0b21a860d9529bf9cf84521fd969fb6f8030059f9b51c39a6d9cb4982a39969" exitCode=0 Nov 02 21:19:19.484640 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:19.484593 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" event={"ID":"db61ea2f-e4f6-4204-b5e2-e047fa0efe02","Type":"ContainerDied","Data":"b0b21a860d9529bf9cf84521fd969fb6f8030059f9b51c39a6d9cb4982a39969"} Nov 02 21:19:20.632202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.632174 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:20.719963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.719924 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-openshift-ca\") pod \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " Nov 02 21:19:20.720160 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.720017 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-kube-api-access-hnbml\" (UniqueName: \"kubernetes.io/projected/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-kube-api-access-hnbml\") pod \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " Nov 02 21:19:20.720160 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.720061 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-xks-ca\") pod \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\" (UID: \"db61ea2f-e4f6-4204-b5e2-e047fa0efe02\") " Nov 02 21:19:20.720387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.720358 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-openshift-ca" pod "db61ea2f-e4f6-4204-b5e2-e047fa0efe02" (UID: "db61ea2f-e4f6-4204-b5e2-e047fa0efe02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:19:20.720539 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.720510 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:20.722313 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.722274 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-xks-ca" pod "db61ea2f-e4f6-4204-b5e2-e047fa0efe02" (UID: "db61ea2f-e4f6-4204-b5e2-e047fa0efe02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:19:20.722419 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.722378 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-kube-api-access-hnbml" pod "db61ea2f-e4f6-4204-b5e2-e047fa0efe02" (UID: "db61ea2f-e4f6-4204-b5e2-e047fa0efe02"). InnerVolumeSpecName "kube-api-access-hnbml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:20.821750 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.821704 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbml\" (UniqueName: \"kubernetes.io/projected/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-kube-api-access-hnbml\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:20.821750 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:20.821743 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db61ea2f-e4f6-4204-b5e2-e047fa0efe02-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:21.497332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:21.497277 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" event={"ID":"db61ea2f-e4f6-4204-b5e2-e047fa0efe02","Type":"ContainerDied","Data":"b720da4481140ad72826909230ed0403c4db38514d2742b229bd4566b8267430"} Nov 02 21:19:21.497332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:21.497319 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9" Nov 02 21:19:21.497332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:21.497343 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b720da4481140ad72826909230ed0403c4db38514d2742b229bd4566b8267430" Nov 02 21:19:22.695764 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:19:22.695726 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:19:26.727301 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:26.727243 5800 scope.go:122] "RemoveContainer" containerID="387ba628a2484c45fc4aefa5642edc70b454add5abe91aff67ff4608044741ec" Nov 02 21:19:39.048873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:39.048841 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9"] Nov 02 21:19:39.094849 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:39.094814 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-056961db-tr9p9"] Nov 02 21:19:39.619942 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:39.619908 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db61ea2f-e4f6-4204-b5e2-e047fa0efe02" path="/var/lib/kubelet/pods/db61ea2f-e4f6-4204-b5e2-e047fa0efe02/volumes" Nov 02 21:19:42.048847 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:42.048810 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs"] Nov 02 21:19:42.095027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:42.094986 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e023d5a-ghlvs"] Nov 02 21:19:43.620204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:43.620165 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74350ac-e8ba-407e-b135-78d8da416b7d" path="/var/lib/kubelet/pods/b74350ac-e8ba-407e-b135-78d8da416b7d/volumes" Nov 02 21:19:52.696209 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:19:52.696173 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:19:54.042800 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:54.042763 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4"] Nov 02 21:19:54.090271 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:54.090231 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b449729-pnsq4"] Nov 02 21:19:55.620926 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:19:55.620889 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5508af2-b09c-4efa-875d-1c86f3c23221" path="/var/lib/kubelet/pods/e5508af2-b09c-4efa-875d-1c86f3c23221/volumes" Nov 02 21:20:09.276659 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.276568 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82"] Nov 02 21:20:09.277161 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.277148 5800 cpu_manager.go:414] "containerMap: removing container" podUID="db61ea2f-e4f6-4204-b5e2-e047fa0efe02" containerName="revoke-keys" Nov 02 21:20:09.277216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.277163 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db61ea2f-e4f6-4204-b5e2-e047fa0efe02" containerName="revoke-keys" Nov 02 21:20:09.277258 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.277232 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="db61ea2f-e4f6-4204-b5e2-e047fa0efe02" containerName="revoke-keys" Nov 02 21:20:09.280429 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.280413 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.375552 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.375514 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-phngj\"" Nov 02 21:20:09.378055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378024 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378159 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378227 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378172 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82"] Nov 02 21:20:09.378273 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378242 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d44f90-b9db-4554-9111-75088930d250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378328 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378293 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378373 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378331 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378465 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378442 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d44f90-b9db-4554-9111-75088930d250-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378549 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378532 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwb59\" (UniqueName: \"kubernetes.io/projected/e7d44f90-b9db-4554-9111-75088930d250-kube-api-access-wwb59\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378566 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d44f90-b9db-4554-9111-75088930d250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.378623 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.378608 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479358 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwb59\" (UniqueName: \"kubernetes.io/projected/e7d44f90-b9db-4554-9111-75088930d250-kube-api-access-wwb59\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479396 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d44f90-b9db-4554-9111-75088930d250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479421 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479462 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479518 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479555 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d44f90-b9db-4554-9111-75088930d250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479598 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479746 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.479955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479783 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d44f90-b9db-4554-9111-75088930d250-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.480092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.479987 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.480092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.480074 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.480196 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.480130 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.480352 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.480335 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.480480 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.480455 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d44f90-b9db-4554-9111-75088930d250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.481829 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.481803 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d44f90-b9db-4554-9111-75088930d250-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.481962 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.481946 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d44f90-b9db-4554-9111-75088930d250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.533058 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.532982 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwb59\" (UniqueName: \"kubernetes.io/projected/e7d44f90-b9db-4554-9111-75088930d250-kube-api-access-wwb59\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.533276 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.533248 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d44f90-b9db-4554-9111-75088930d250-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-2sf82\" (UID: \"e7d44f90-b9db-4554-9111-75088930d250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:09.593907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:09.593869 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:10.010475 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:10.010439 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82"] Nov 02 21:20:10.046512 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:20:10.046468 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d44f90_b9db_4554_9111_75088930d250.slice/crio-699fbf35843aa96b3ffd62067866936f8e53d3dd65fb42246824f93529ab2859 WatchSource:0}: Error finding container 699fbf35843aa96b3ffd62067866936f8e53d3dd65fb42246824f93529ab2859: Status 404 returned error can't find the container with id 699fbf35843aa96b3ffd62067866936f8e53d3dd65fb42246824f93529ab2859 Nov 02 21:20:10.048569 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:10.048535 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:10.048907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:10.048602 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:10.048907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:10.048632 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:20:10.778974 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:10.778934 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" event={"ID":"e7d44f90-b9db-4554-9111-75088930d250","Type":"ContainerStarted","Data":"122398dc468116707375df15a478ae4ec1b0557492538a17d22c3011070205c4"} Nov 02 21:20:10.779439 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:10.779036 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" event={"ID":"e7d44f90-b9db-4554-9111-75088930d250","Type":"ContainerStarted","Data":"699fbf35843aa96b3ffd62067866936f8e53d3dd65fb42246824f93529ab2859"} Nov 02 21:20:10.998214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:10.998153 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podStartSLOduration=1.998134811 podStartE2EDuration="1.998134811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:10.951626266 +0000 UTC m=+5509.468398852" watchObservedRunningTime="2026-11-02 21:20:10.998134811 +0000 UTC m=+5509.514907395" Nov 02 21:20:11.594410 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:11.594366 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:11.596105 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:11.596075 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 02 21:20:11.596268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:11.596137 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 02 21:20:12.594249 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:12.594201 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 02 21:20:12.594698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:12.594257 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 02 21:20:13.594560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:13.594522 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 02 21:20:13.594973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:13.594587 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 02 21:20:14.594511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:14.594468 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 02 21:20:14.594847 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:14.594532 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 02 21:20:15.594515 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:15.594480 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 02 21:20:15.594709 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:15.594533 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 02 21:20:16.594472 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:16.594437 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" start-of-body= Nov 02 21:20:16.594967 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:16.594496 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": dial tcp 10.132.1.102:15021: connect: connection refused" Nov 02 21:20:18.595876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:18.595828 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:20:18.596466 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:18.595963 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:20:19.637483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:19.637440 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:20:19.637905 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:19.637514 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:20:20.637052 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:20.637020 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:20.637316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:20.637297 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:20.637962 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:20.637942 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" Nov 02 21:20:22.695845 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:20:22.695804 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:20:26.758226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:26.758189 5800 scope.go:122] "RemoveContainer" containerID="827193f21e1d4bd6399c6cff161c9cd6c3ec270d91b1ee173bbc7326da96a2fc" Nov 02 21:20:45.049202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:45.049143 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng"] Nov 02 21:20:45.095161 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:45.095126 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cd95d5-pwdng"] Nov 02 21:20:45.620863 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:45.620824 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a" path="/var/lib/kubelet/pods/cb46e3bb-fd96-4d20-b8f0-7d4dae45e12a/volumes" Nov 02 21:20:52.699901 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:20:52.699862 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:20:57.042224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:57.042171 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk"] Nov 02 21:20:57.088523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:57.088490 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbd82c55-jf9sk"] Nov 02 21:20:57.620483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:20:57.620449 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1be430-a931-4237-937e-2597f724d57d" path="/var/lib/kubelet/pods/cc1be430-a931-4237-937e-2597f724d57d/volumes" Nov 02 21:21:08.043851 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:08.043814 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd"] Nov 02 21:21:08.090191 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:08.090146 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7afdf34-kgrfd"] Nov 02 21:21:09.621195 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:09.621158 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efdc0d8e-685b-4ae0-ab76-709b2d3b79f4" path="/var/lib/kubelet/pods/efdc0d8e-685b-4ae0-ab76-709b2d3b79f4/volumes" Nov 02 21:21:22.695549 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:21:22.695512 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:21:26.782681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:26.782637 5800 scope.go:122] "RemoveContainer" containerID="ef17446007badd2e7b8438c5239b42f5b16a04718ee5b2777e7662ac7a8e3e79" Nov 02 21:21:26.793829 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:26.793804 5800 scope.go:122] "RemoveContainer" containerID="8670c6b38e9c1f5f5c269903a9a7fd42318a595f647a9affa627785d2bbd584e" Nov 02 21:21:30.895651 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.895565 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8"] Nov 02 21:21:30.900728 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.900706 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:30.950013 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.949978 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:21:30.950013 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.949978 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:21:30.952247 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.952221 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8"] Nov 02 21:21:30.984575 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.984537 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:30.984734 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.984594 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxdpz\" (UniqueName: \"kubernetes.io/projected/ca583944-4a9a-4a82-8034-ebf3bbdb045b-kube-api-access-bxdpz\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:30.984780 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.984725 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:30.995451 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.995422 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:21:30.995581 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:30.995464 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:21:31.086370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.086330 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:31.086591 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.086408 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:31.086591 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.086438 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxdpz\" (UniqueName: \"kubernetes.io/projected/ca583944-4a9a-4a82-8034-ebf3bbdb045b-kube-api-access-bxdpz\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:31.087087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.087058 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:31.088849 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.088830 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:31.140033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.140003 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxdpz\" (UniqueName: \"kubernetes.io/projected/ca583944-4a9a-4a82-8034-ebf3bbdb045b-kube-api-access-bxdpz\") pod \"maas-api-revoke-sub-f03629db-qmmq8\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:31.214345 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.214226 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:31.584381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:31.584348 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8"] Nov 02 21:21:31.586453 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:21:31.586427 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca583944_4a9a_4a82_8034_ebf3bbdb045b.slice/crio-cb4fe6bfe3ba23b7097a31b0d6c5a5539dec722a2a9e010d2d0311ec380f452f WatchSource:0}: Error finding container cb4fe6bfe3ba23b7097a31b0d6c5a5539dec722a2a9e010d2d0311ec380f452f: Status 404 returned error can't find the container with id cb4fe6bfe3ba23b7097a31b0d6c5a5539dec722a2a9e010d2d0311ec380f452f Nov 02 21:21:32.247604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:32.247505 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" event={"ID":"ca583944-4a9a-4a82-8034-ebf3bbdb045b","Type":"ContainerStarted","Data":"9e2364fe3b4e7cdcea9ff032db88d11f1e21766bc05ac9496f4f6650dd6ed2b5"} Nov 02 21:21:32.247604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:32.247574 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" event={"ID":"ca583944-4a9a-4a82-8034-ebf3bbdb045b","Type":"ContainerStarted","Data":"cb4fe6bfe3ba23b7097a31b0d6c5a5539dec722a2a9e010d2d0311ec380f452f"} Nov 02 21:21:32.353920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:32.353869 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" podStartSLOduration=2.353854644 podStartE2EDuration="2.353854644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:32.352744617 +0000 UTC m=+5590.869517200" watchObservedRunningTime="2026-11-02 21:21:32.353854644 +0000 UTC m=+5590.870627223" Nov 02 21:21:33.255268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:33.255232 5800 generic.go:356] "Generic (PLEG): container finished" podID="ca583944-4a9a-4a82-8034-ebf3bbdb045b" containerID="9e2364fe3b4e7cdcea9ff032db88d11f1e21766bc05ac9496f4f6650dd6ed2b5" exitCode=0 Nov 02 21:21:33.255711 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:33.255275 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" event={"ID":"ca583944-4a9a-4a82-8034-ebf3bbdb045b","Type":"ContainerDied","Data":"9e2364fe3b4e7cdcea9ff032db88d11f1e21766bc05ac9496f4f6650dd6ed2b5"} Nov 02 21:21:34.393202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.393175 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:34.523106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.523027 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca583944-4a9a-4a82-8034-ebf3bbdb045b-kube-api-access-bxdpz\" (UniqueName: \"kubernetes.io/projected/ca583944-4a9a-4a82-8034-ebf3bbdb045b-kube-api-access-bxdpz\") pod \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " Nov 02 21:21:34.523307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.523119 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-openshift-ca\") pod \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " Nov 02 21:21:34.523307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.523167 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-xks-ca\") pod \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\" (UID: \"ca583944-4a9a-4a82-8034-ebf3bbdb045b\") " Nov 02 21:21:34.523513 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.523485 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-openshift-ca" pod "ca583944-4a9a-4a82-8034-ebf3bbdb045b" (UID: "ca583944-4a9a-4a82-8034-ebf3bbdb045b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:34.523648 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.523593 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:34.525327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.525294 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca583944-4a9a-4a82-8034-ebf3bbdb045b-kube-api-access-bxdpz" pod "ca583944-4a9a-4a82-8034-ebf3bbdb045b" (UID: "ca583944-4a9a-4a82-8034-ebf3bbdb045b"). InnerVolumeSpecName "kube-api-access-bxdpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:34.525598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.525577 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-xks-ca" pod "ca583944-4a9a-4a82-8034-ebf3bbdb045b" (UID: "ca583944-4a9a-4a82-8034-ebf3bbdb045b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:34.624197 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.624161 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca583944-4a9a-4a82-8034-ebf3bbdb045b-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:34.624197 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:34.624188 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxdpz\" (UniqueName: \"kubernetes.io/projected/ca583944-4a9a-4a82-8034-ebf3bbdb045b-kube-api-access-bxdpz\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:35.267139 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:35.267102 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" Nov 02 21:21:35.267139 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:35.267127 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8" event={"ID":"ca583944-4a9a-4a82-8034-ebf3bbdb045b","Type":"ContainerDied","Data":"cb4fe6bfe3ba23b7097a31b0d6c5a5539dec722a2a9e010d2d0311ec380f452f"} Nov 02 21:21:35.267394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:35.267175 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4fe6bfe3ba23b7097a31b0d6c5a5539dec722a2a9e010d2d0311ec380f452f" Nov 02 21:21:49.826854 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.826820 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv"] Nov 02 21:21:49.827309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.827294 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ca583944-4a9a-4a82-8034-ebf3bbdb045b" containerName="revoke-keys" Nov 02 21:21:49.827364 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.827311 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca583944-4a9a-4a82-8034-ebf3bbdb045b" containerName="revoke-keys" Nov 02 21:21:49.827406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.827394 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca583944-4a9a-4a82-8034-ebf3bbdb045b" containerName="revoke-keys" Nov 02 21:21:49.838865 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.838830 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:49.884595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.884562 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv"] Nov 02 21:21:49.928327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.928270 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:21:49.928540 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.928348 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:21:49.933794 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.933769 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:21:49.933923 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.933849 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:21:49.980296 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.980261 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:49.980491 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.980417 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8wd\" (UniqueName: \"kubernetes.io/projected/7d3351ec-bd1e-4d7d-8412-62671a9991f7-kube-api-access-kx8wd\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:49.980571 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:49.980550 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.081893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.081806 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.081893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.081857 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.082091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.081912 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8wd\" (UniqueName: \"kubernetes.io/projected/7d3351ec-bd1e-4d7d-8412-62671a9991f7-kube-api-access-kx8wd\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.082598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.082573 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.084252 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.084224 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.135482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.135442 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8wd\" (UniqueName: \"kubernetes.io/projected/7d3351ec-bd1e-4d7d-8412-62671a9991f7-kube-api-access-kx8wd\") pod \"maas-api-revoke-sub-0165d7d2-6v4mv\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.154116 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.154082 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:50.522832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:50.522793 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv"] Nov 02 21:21:50.524575 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:21:50.524550 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3351ec_bd1e_4d7d_8412_62671a9991f7.slice/crio-acd84ac3fde24ec526ff448f4e85e5025bc3dab60d037e8ff12ddd8992b2a21d WatchSource:0}: Error finding container acd84ac3fde24ec526ff448f4e85e5025bc3dab60d037e8ff12ddd8992b2a21d: Status 404 returned error can't find the container with id acd84ac3fde24ec526ff448f4e85e5025bc3dab60d037e8ff12ddd8992b2a21d Nov 02 21:21:51.361074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:51.361039 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" event={"ID":"7d3351ec-bd1e-4d7d-8412-62671a9991f7","Type":"ContainerStarted","Data":"9f32161a60e24437774dda8346e4feb13b6e90a731a323eb7432889ad33f5e60"} Nov 02 21:21:51.361492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:51.361098 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" event={"ID":"7d3351ec-bd1e-4d7d-8412-62671a9991f7","Type":"ContainerStarted","Data":"acd84ac3fde24ec526ff448f4e85e5025bc3dab60d037e8ff12ddd8992b2a21d"} Nov 02 21:21:51.467817 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:51.467754 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" podStartSLOduration=2.467737893 podStartE2EDuration="2.467737893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:21:51.465585474 +0000 UTC m=+5609.982358060" watchObservedRunningTime="2026-11-02 21:21:51.467737893 +0000 UTC m=+5609.984510477" Nov 02 21:21:52.367887 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:52.367849 5800 generic.go:356] "Generic (PLEG): container finished" podID="7d3351ec-bd1e-4d7d-8412-62671a9991f7" containerID="9f32161a60e24437774dda8346e4feb13b6e90a731a323eb7432889ad33f5e60" exitCode=0 Nov 02 21:21:52.368322 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:52.367927 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" event={"ID":"7d3351ec-bd1e-4d7d-8412-62671a9991f7","Type":"ContainerDied","Data":"9f32161a60e24437774dda8346e4feb13b6e90a731a323eb7432889ad33f5e60"} Nov 02 21:21:52.696497 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:21:52.696412 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:21:53.519777 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.519749 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:53.615742 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.615711 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-openshift-ca\") pod \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " Nov 02 21:21:53.615742 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.615751 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-xks-ca\") pod \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " Nov 02 21:21:53.616036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.615898 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3351ec-bd1e-4d7d-8412-62671a9991f7-kube-api-access-kx8wd\" (UniqueName: \"kubernetes.io/projected/7d3351ec-bd1e-4d7d-8412-62671a9991f7-kube-api-access-kx8wd\") pod \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\" (UID: \"7d3351ec-bd1e-4d7d-8412-62671a9991f7\") " Nov 02 21:21:53.616097 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.616055 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-openshift-ca" pod "7d3351ec-bd1e-4d7d-8412-62671a9991f7" (UID: "7d3351ec-bd1e-4d7d-8412-62671a9991f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:21:53.616220 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.616200 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:53.618082 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.618046 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3351ec-bd1e-4d7d-8412-62671a9991f7-kube-api-access-kx8wd" pod "7d3351ec-bd1e-4d7d-8412-62671a9991f7" (UID: "7d3351ec-bd1e-4d7d-8412-62671a9991f7"). InnerVolumeSpecName "kube-api-access-kx8wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:21:53.618359 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.618339 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-xks-ca" pod "7d3351ec-bd1e-4d7d-8412-62671a9991f7" (UID: "7d3351ec-bd1e-4d7d-8412-62671a9991f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:21:53.717124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.717089 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx8wd\" (UniqueName: \"kubernetes.io/projected/7d3351ec-bd1e-4d7d-8412-62671a9991f7-kube-api-access-kx8wd\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:53.717124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:53.717123 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3351ec-bd1e-4d7d-8412-62671a9991f7-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:21:54.380835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:54.380802 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" Nov 02 21:21:54.381024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:54.380841 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv" event={"ID":"7d3351ec-bd1e-4d7d-8412-62671a9991f7","Type":"ContainerDied","Data":"acd84ac3fde24ec526ff448f4e85e5025bc3dab60d037e8ff12ddd8992b2a21d"} Nov 02 21:21:54.381024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:21:54.380890 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd84ac3fde24ec526ff448f4e85e5025bc3dab60d037e8ff12ddd8992b2a21d" Nov 02 21:22:01.455137 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.455102 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw"] Nov 02 21:22:01.455606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.455533 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7d3351ec-bd1e-4d7d-8412-62671a9991f7" containerName="revoke-keys" Nov 02 21:22:01.455606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.455545 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3351ec-bd1e-4d7d-8412-62671a9991f7" containerName="revoke-keys" Nov 02 21:22:01.455703 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.455624 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3351ec-bd1e-4d7d-8412-62671a9991f7" containerName="revoke-keys" Nov 02 21:22:01.458587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.458566 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.474149 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.474113 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcqd\" (UniqueName: \"kubernetes.io/projected/c8f2074c-c7f4-4d71-bd27-f1023234c93e-kube-api-access-nwcqd\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.474312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.474154 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.474366 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.474275 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.554990 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.554952 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:22:01.554990 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.554981 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:22:01.555321 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.555300 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:22:01.555384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.555369 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:22:01.557342 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.557319 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw"] Nov 02 21:22:01.575412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.575383 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcqd\" (UniqueName: \"kubernetes.io/projected/c8f2074c-c7f4-4d71-bd27-f1023234c93e-kube-api-access-nwcqd\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.575589 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.575417 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.575589 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.575455 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.576216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.576188 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.577884 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.577864 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.628867 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.628831 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcqd\" (UniqueName: \"kubernetes.io/projected/c8f2074c-c7f4-4d71-bd27-f1023234c93e-kube-api-access-nwcqd\") pod \"maas-api-revoke-sub-6b19338f-qqpkw\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:01.771610 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:01.771578 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:02.136572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:02.136541 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw"] Nov 02 21:22:02.139394 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:22:02.139357 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f2074c_c7f4_4d71_bd27_f1023234c93e.slice/crio-ac8520891a5c1b9b3b9a0936244c37a714b0469a707da388da00c303ffb4e233 WatchSource:0}: Error finding container ac8520891a5c1b9b3b9a0936244c37a714b0469a707da388da00c303ffb4e233: Status 404 returned error can't find the container with id ac8520891a5c1b9b3b9a0936244c37a714b0469a707da388da00c303ffb4e233 Nov 02 21:22:02.436649 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:02.436528 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" event={"ID":"c8f2074c-c7f4-4d71-bd27-f1023234c93e","Type":"ContainerStarted","Data":"ac8520891a5c1b9b3b9a0936244c37a714b0469a707da388da00c303ffb4e233"} Nov 02 21:22:03.443639 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:03.443604 5800 generic.go:356] "Generic (PLEG): container finished" podID="c8f2074c-c7f4-4d71-bd27-f1023234c93e" containerID="4215162b4cebd4f77e877cee8c64655a194241763fd8c77d518d70f03e53c2c2" exitCode=0 Nov 02 21:22:03.444063 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:03.443692 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" event={"ID":"c8f2074c-c7f4-4d71-bd27-f1023234c93e","Type":"ContainerDied","Data":"4215162b4cebd4f77e877cee8c64655a194241763fd8c77d518d70f03e53c2c2"} Nov 02 21:22:04.616990 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.616963 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:04.708558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.708471 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-openshift-ca\") pod \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " Nov 02 21:22:04.708731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.708569 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-xks-ca\") pod \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " Nov 02 21:22:04.708731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.708606 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f2074c-c7f4-4d71-bd27-f1023234c93e-kube-api-access-nwcqd\" (UniqueName: \"kubernetes.io/projected/c8f2074c-c7f4-4d71-bd27-f1023234c93e-kube-api-access-nwcqd\") pod \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\" (UID: \"c8f2074c-c7f4-4d71-bd27-f1023234c93e\") " Nov 02 21:22:04.708838 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.708814 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-openshift-ca" pod "c8f2074c-c7f4-4d71-bd27-f1023234c93e" (UID: "c8f2074c-c7f4-4d71-bd27-f1023234c93e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:22:04.708956 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.708929 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:22:04.710666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.710645 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-xks-ca" pod "c8f2074c-c7f4-4d71-bd27-f1023234c93e" (UID: "c8f2074c-c7f4-4d71-bd27-f1023234c93e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:22:04.710779 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.710762 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f2074c-c7f4-4d71-bd27-f1023234c93e-kube-api-access-nwcqd" pod "c8f2074c-c7f4-4d71-bd27-f1023234c93e" (UID: "c8f2074c-c7f4-4d71-bd27-f1023234c93e"). InnerVolumeSpecName "kube-api-access-nwcqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:22:04.809678 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.809637 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f2074c-c7f4-4d71-bd27-f1023234c93e-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:22:04.809678 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:04.809668 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcqd\" (UniqueName: \"kubernetes.io/projected/c8f2074c-c7f4-4d71-bd27-f1023234c93e-kube-api-access-nwcqd\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:22:05.456668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:05.456638 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" Nov 02 21:22:05.456668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:05.456659 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw" event={"ID":"c8f2074c-c7f4-4d71-bd27-f1023234c93e","Type":"ContainerDied","Data":"ac8520891a5c1b9b3b9a0936244c37a714b0469a707da388da00c303ffb4e233"} Nov 02 21:22:05.456955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:05.456703 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac8520891a5c1b9b3b9a0936244c37a714b0469a707da388da00c303ffb4e233" Nov 02 21:22:10.199408 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.199370 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm"] Nov 02 21:22:10.199807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.199786 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c8f2074c-c7f4-4d71-bd27-f1023234c93e" containerName="revoke-keys" Nov 02 21:22:10.199807 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.199799 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f2074c-c7f4-4d71-bd27-f1023234c93e" containerName="revoke-keys" Nov 02 21:22:10.199913 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.199859 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f2074c-c7f4-4d71-bd27-f1023234c93e" containerName="revoke-keys" Nov 02 21:22:10.202907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.202890 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.246008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.245970 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5"] Nov 02 21:22:10.249573 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.249557 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.260545 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260520 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260556 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260638 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260665 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260689 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260761 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87h4x\" (UniqueName: \"kubernetes.io/projected/48b27c1e-b9de-4d19-9b28-656ee6a0f697-kube-api-access-87h4x\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260816 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260961 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260878 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.260961 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.260930 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.298325 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.298266 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm"] Nov 02 21:22:10.352701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.352663 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5"] Nov 02 21:22:10.362303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362237 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362306 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362567 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362404 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.362567 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362424 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362567 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362440 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.362567 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362459 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87h4x\" (UniqueName: \"kubernetes.io/projected/48b27c1e-b9de-4d19-9b28-656ee6a0f697-kube-api-access-87h4x\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362567 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362536 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362569 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362610 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362631 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362754 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362767 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362789 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362742 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.362845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362825 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.363207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362859 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.363207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362938 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khsc5\" (UniqueName: \"kubernetes.io/projected/6eaf89b5-5d8f-4515-9c00-cef33ea31033-kube-api-access-khsc5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.363207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362945 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.363207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.362966 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.363207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.363016 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.363207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.363047 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.363513 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.363375 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.363549 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.363504 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.365199 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.365174 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.365513 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.365491 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.418786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.418753 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87h4x\" (UniqueName: \"kubernetes.io/projected/48b27c1e-b9de-4d19-9b28-656ee6a0f697-kube-api-access-87h4x\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.419206 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.419181 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b27c1e-b9de-4d19-9b28-656ee6a0f697-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-7h7jm\" (UID: \"48b27c1e-b9de-4d19-9b28-656ee6a0f697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.463787 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.463701 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.463787 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.463738 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.463787 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.463761 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.463887 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.463952 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.463987 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464020 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464067 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464114 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsc5\" (UniqueName: \"kubernetes.io/projected/6eaf89b5-5d8f-4515-9c00-cef33ea31033-kube-api-access-khsc5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464126 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464187 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464221 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464619 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464475 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.464780 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.464761 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.466431 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.466409 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.466517 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.466504 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.515530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.515497 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:10.517328 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.517274 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaf89b5-5d8f-4515-9c00-cef33ea31033-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.517456 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.517425 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsc5\" (UniqueName: \"kubernetes.io/projected/6eaf89b5-5d8f-4515-9c00-cef33ea31033-kube-api-access-khsc5\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-h2gr5\" (UID: \"6eaf89b5-5d8f-4515-9c00-cef33ea31033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.562070 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.562025 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:10.931006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.930941 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm"] Nov 02 21:22:10.934869 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:22:10.934837 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b27c1e_b9de_4d19_9b28_656ee6a0f697.slice/crio-19007543c41d5b26d4afd3308cc007564d1218362d8bc069a78f543c5695e866 WatchSource:0}: Error finding container 19007543c41d5b26d4afd3308cc007564d1218362d8bc069a78f543c5695e866: Status 404 returned error can't find the container with id 19007543c41d5b26d4afd3308cc007564d1218362d8bc069a78f543c5695e866 Nov 02 21:22:10.936778 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.936743 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:10.936858 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.936804 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:10.936858 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.936836 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:10.980606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.980580 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5"] Nov 02 21:22:10.981499 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:22:10.981471 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaf89b5_5d8f_4515_9c00_cef33ea31033.slice/crio-dc6b1b98effb1965248d1a493b23b4858afde349942e6092d67101e38902afcf WatchSource:0}: Error finding container dc6b1b98effb1965248d1a493b23b4858afde349942e6092d67101e38902afcf: Status 404 returned error can't find the container with id dc6b1b98effb1965248d1a493b23b4858afde349942e6092d67101e38902afcf Nov 02 21:22:10.984134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.984104 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:10.984252 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.984178 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:10.984252 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:10.984224 5800 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:22:11.493046 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.493004 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" event={"ID":"6eaf89b5-5d8f-4515-9c00-cef33ea31033","Type":"ContainerStarted","Data":"9737b47c8da8b78aeff0dd8c79c6fcda3cd7e1bf55562073efb9a8e583a9b7a3"} Nov 02 21:22:11.493737 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.493698 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" event={"ID":"6eaf89b5-5d8f-4515-9c00-cef33ea31033","Type":"ContainerStarted","Data":"dc6b1b98effb1965248d1a493b23b4858afde349942e6092d67101e38902afcf"} Nov 02 21:22:11.495108 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.495084 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" event={"ID":"48b27c1e-b9de-4d19-9b28-656ee6a0f697","Type":"ContainerStarted","Data":"4e724a55116f90f58e4df4a8db1de0222ce491e21699c6cbc15179568ae1716f"} Nov 02 21:22:11.495237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.495126 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" event={"ID":"48b27c1e-b9de-4d19-9b28-656ee6a0f697","Type":"ContainerStarted","Data":"19007543c41d5b26d4afd3308cc007564d1218362d8bc069a78f543c5695e866"} Nov 02 21:22:11.516046 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.516009 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:11.517731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.517697 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:11.517915 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.517747 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:11.562561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.562517 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:11.605211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.605150 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podStartSLOduration=1.60513717 podStartE2EDuration="1.60513717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:11.603535458 +0000 UTC m=+5630.120308042" watchObservedRunningTime="2026-11-02 21:22:11.60513717 +0000 UTC m=+5630.121909822" Nov 02 21:22:11.719686 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:11.719367 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podStartSLOduration=1.7193485339999999 podStartE2EDuration="1.719348534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:11.718913881 +0000 UTC m=+5630.235686467" watchObservedRunningTime="2026-11-02 21:22:11.719348534 +0000 UTC m=+5630.236121139" Nov 02 21:22:12.516051 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:12.516015 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:12.516496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:12.516076 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:12.587246 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:12.587208 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 02 21:22:12.591944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:12.591902 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 02 21:22:13.516176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:13.516146 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:13.516693 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:13.516200 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:13.563307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:13.562432 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 02 21:22:13.563307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:13.562498 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 02 21:22:14.516532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:14.516484 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:14.516981 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:14.516565 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:14.563850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:14.563719 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 02 21:22:14.563850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:14.563789 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 02 21:22:15.516327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:15.516254 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:15.516690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:15.516642 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:15.563877 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:15.563837 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 02 21:22:15.564208 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:15.564160 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 02 21:22:16.516693 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:16.516648 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:16.517241 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:16.517192 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:16.565989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:16.565954 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 02 21:22:16.566346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:16.566314 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 02 21:22:17.516824 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:17.516786 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:17.517214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:17.516853 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:17.562355 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:17.562319 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 02 21:22:17.562506 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:17.562385 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 02 21:22:18.516547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:18.516502 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 02 21:22:18.516743 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:18.516566 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 02 21:22:18.563213 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:18.563163 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 02 21:22:18.563700 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:18.563232 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 02 21:22:20.517489 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:20.517445 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:20.518136 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:20.518096 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:20.603492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:20.603449 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:20.603689 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:20.603522 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:21.559521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:21.559481 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:21.560189 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:21.560116 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:21.653347 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:21.649376 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:21.653347 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:21.649441 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:22.271303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.270274 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:22.272710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.271687 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:22.272710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.272613 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" Nov 02 21:22:22.567471 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.567362 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:22:22.568043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.568001 5800 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:22:22.722927 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:22:22.722880 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:22:22.741832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.741788 5800 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:22.743450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.743428 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:22.747690 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:22.747667 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" Nov 02 21:22:26.821245 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:26.821205 5800 scope.go:122] "RemoveContainer" containerID="bd39124d60a6905f00b87bc60b40f8832708f30713f9adb59525316564a1e1c0" Nov 02 21:22:26.830483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:26.830455 5800 scope.go:122] "RemoveContainer" containerID="35528566236c050c88b82f8f5dd0289f1224596f2fef730fd1c06a691cf77dc9" Nov 02 21:22:26.839136 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:26.839112 5800 scope.go:122] "RemoveContainer" containerID="fc68d40688624900a5a8092b21e917588baa5673637fa313402248f055b08d88" Nov 02 21:22:52.694775 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:22:52.694731 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:22:58.048587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:58.045243 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw"] Nov 02 21:22:58.089276 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:58.089239 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67db470-88vxw"] Nov 02 21:22:59.620523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:22:59.620486 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5796aa16-b57e-4619-ae6f-97f820af19be" path="/var/lib/kubelet/pods/5796aa16-b57e-4619-ae6f-97f820af19be/volumes" Nov 02 21:23:08.328872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:08.328448 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:23:08.328872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:08.328550 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:23:10.042786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:10.042737 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn"] Nov 02 21:23:10.088981 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:10.088949 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-605538c6-wxgwn"] Nov 02 21:23:11.620793 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:11.620758 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0ddf21-c52f-49fe-89af-482bbddfc0be" path="/var/lib/kubelet/pods/2d0ddf21-c52f-49fe-89af-482bbddfc0be/volumes" Nov 02 21:23:16.195671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.195635 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v"] Nov 02 21:23:16.199528 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.199510 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.250984 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.250947 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v"] Nov 02 21:23:16.280061 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.280020 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6wp\" (UniqueName: \"kubernetes.io/projected/d2f50641-f653-4697-b325-d09cbedb5594-kube-api-access-vf6wp\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.280061 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.280074 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.280359 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.280225 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.293959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.293925 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:23:16.293959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.293953 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:23:16.294146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.293965 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:23:16.294688 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.294672 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:23:16.380957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.380921 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.381164 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.381008 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6wp\" (UniqueName: \"kubernetes.io/projected/d2f50641-f653-4697-b325-d09cbedb5594-kube-api-access-vf6wp\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.381164 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.381037 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.382056 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.382021 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.383452 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.383431 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.434982 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.434947 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6wp\" (UniqueName: \"kubernetes.io/projected/d2f50641-f653-4697-b325-d09cbedb5594-kube-api-access-vf6wp\") pod \"maas-api-revoke-sub-a3035529-kzx8v\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.512724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.512694 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:16.877266 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.877240 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v"] Nov 02 21:23:16.879322 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:23:16.879292 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f50641_f653_4697_b325_d09cbedb5594.slice/crio-09fc7bf0b094afa9375929c6908853d1777cf274a2f2eccf7821dff324d21745 WatchSource:0}: Error finding container 09fc7bf0b094afa9375929c6908853d1777cf274a2f2eccf7821dff324d21745: Status 404 returned error can't find the container with id 09fc7bf0b094afa9375929c6908853d1777cf274a2f2eccf7821dff324d21745 Nov 02 21:23:16.944630 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:16.944595 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" event={"ID":"d2f50641-f653-4697-b325-d09cbedb5594","Type":"ContainerStarted","Data":"09fc7bf0b094afa9375929c6908853d1777cf274a2f2eccf7821dff324d21745"} Nov 02 21:23:17.952259 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:17.952163 5800 generic.go:356] "Generic (PLEG): container finished" podID="d2f50641-f653-4697-b325-d09cbedb5594" containerID="db7496c5c40d79f6d39524e278db372a0c4377f4e00a89bab3aa1c5cb9de8b48" exitCode=0 Nov 02 21:23:17.952681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:17.952247 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" event={"ID":"d2f50641-f653-4697-b325-d09cbedb5594","Type":"ContainerDied","Data":"db7496c5c40d79f6d39524e278db372a0c4377f4e00a89bab3aa1c5cb9de8b48"} Nov 02 21:23:19.106321 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.106297 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:19.211253 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.211155 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-xks-ca\") pod \"d2f50641-f653-4697-b325-d09cbedb5594\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " Nov 02 21:23:19.211478 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.211370 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f50641-f653-4697-b325-d09cbedb5594-kube-api-access-vf6wp\" (UniqueName: \"kubernetes.io/projected/d2f50641-f653-4697-b325-d09cbedb5594-kube-api-access-vf6wp\") pod \"d2f50641-f653-4697-b325-d09cbedb5594\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " Nov 02 21:23:19.211587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.211555 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-openshift-ca\") pod \"d2f50641-f653-4697-b325-d09cbedb5594\" (UID: \"d2f50641-f653-4697-b325-d09cbedb5594\") " Nov 02 21:23:19.212321 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.212243 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-openshift-ca" pod "d2f50641-f653-4697-b325-d09cbedb5594" (UID: "d2f50641-f653-4697-b325-d09cbedb5594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:23:19.214014 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.213991 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-xks-ca" pod "d2f50641-f653-4697-b325-d09cbedb5594" (UID: "d2f50641-f653-4697-b325-d09cbedb5594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:23:19.214595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.214571 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f50641-f653-4697-b325-d09cbedb5594-kube-api-access-vf6wp" pod "d2f50641-f653-4697-b325-d09cbedb5594" (UID: "d2f50641-f653-4697-b325-d09cbedb5594"). InnerVolumeSpecName "kube-api-access-vf6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:23:19.312867 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.312829 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:19.312867 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.312861 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6wp\" (UniqueName: \"kubernetes.io/projected/d2f50641-f653-4697-b325-d09cbedb5594-kube-api-access-vf6wp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:19.312867 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.312871 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f50641-f653-4697-b325-d09cbedb5594-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:19.964991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.964897 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" Nov 02 21:23:19.964991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.964936 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v" event={"ID":"d2f50641-f653-4697-b325-d09cbedb5594","Type":"ContainerDied","Data":"09fc7bf0b094afa9375929c6908853d1777cf274a2f2eccf7821dff324d21745"} Nov 02 21:23:19.964991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:19.964988 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09fc7bf0b094afa9375929c6908853d1777cf274a2f2eccf7821dff324d21745" Nov 02 21:23:22.688849 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:22.688818 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:23:22.698069 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:22.698032 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:23:22.699684 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:23:22.699640 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:23:34.483568 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.483523 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8"] Nov 02 21:23:34.484240 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.484210 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d2f50641-f653-4697-b325-d09cbedb5594" containerName="revoke-keys" Nov 02 21:23:34.484240 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.484240 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f50641-f653-4697-b325-d09cbedb5594" containerName="revoke-keys" Nov 02 21:23:34.484500 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.484403 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f50641-f653-4697-b325-d09cbedb5594" containerName="revoke-keys" Nov 02 21:23:34.489402 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.489374 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.550305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.549755 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8"] Nov 02 21:23:34.579619 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.579578 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.579821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.579650 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltr8\" (UniqueName: \"kubernetes.io/projected/d1f4bc4a-625a-4876-b92a-5b820b6d3041-kube-api-access-nltr8\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.579821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.579697 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.585230 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.585199 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:23:34.585406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.585273 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:23:34.585406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.585371 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:23:34.585542 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.585436 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:23:34.680532 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.680478 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltr8\" (UniqueName: \"kubernetes.io/projected/d1f4bc4a-625a-4876-b92a-5b820b6d3041-kube-api-access-nltr8\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.680749 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.680586 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.680749 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.680685 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.681678 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.681644 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.683323 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.683276 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.736109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.736024 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltr8\" (UniqueName: \"kubernetes.io/projected/d1f4bc4a-625a-4876-b92a-5b820b6d3041-kube-api-access-nltr8\") pod \"maas-api-revoke-sub-05ad9e50-lrjm8\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:34.816585 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:34.816543 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:35.204051 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:35.204024 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8"] Nov 02 21:23:35.206478 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:23:35.206448 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f4bc4a_625a_4876_b92a_5b820b6d3041.slice/crio-6b6aacfff8f8a49499218094eac939cb60f6e47b48df4428135045ede7aeb6ad WatchSource:0}: Error finding container 6b6aacfff8f8a49499218094eac939cb60f6e47b48df4428135045ede7aeb6ad: Status 404 returned error can't find the container with id 6b6aacfff8f8a49499218094eac939cb60f6e47b48df4428135045ede7aeb6ad Nov 02 21:23:36.095333 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:36.089906 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" event={"ID":"d1f4bc4a-625a-4876-b92a-5b820b6d3041","Type":"ContainerStarted","Data":"429f7686307a1cea71c9f13efaa43d61a52ae4cd159c723804371fff8746f1ce"} Nov 02 21:23:36.095333 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:36.089982 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" event={"ID":"d1f4bc4a-625a-4876-b92a-5b820b6d3041","Type":"ContainerStarted","Data":"6b6aacfff8f8a49499218094eac939cb60f6e47b48df4428135045ede7aeb6ad"} Nov 02 21:23:36.693130 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:36.693008 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:23:36.693344 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:36.693118 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:23:37.097235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:37.097196 5800 generic.go:356] "Generic (PLEG): container finished" podID="d1f4bc4a-625a-4876-b92a-5b820b6d3041" containerID="429f7686307a1cea71c9f13efaa43d61a52ae4cd159c723804371fff8746f1ce" exitCode=0 Nov 02 21:23:37.097680 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:37.097292 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" event={"ID":"d1f4bc4a-625a-4876-b92a-5b820b6d3041","Type":"ContainerDied","Data":"429f7686307a1cea71c9f13efaa43d61a52ae4cd159c723804371fff8746f1ce"} Nov 02 21:23:38.252903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.252873 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:38.334148 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.334112 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f4bc4a-625a-4876-b92a-5b820b6d3041-kube-api-access-nltr8\" (UniqueName: \"kubernetes.io/projected/d1f4bc4a-625a-4876-b92a-5b820b6d3041-kube-api-access-nltr8\") pod \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " Nov 02 21:23:38.334381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.334271 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-openshift-ca\") pod \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " Nov 02 21:23:38.334381 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.334351 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-xks-ca\") pod \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\" (UID: \"d1f4bc4a-625a-4876-b92a-5b820b6d3041\") " Nov 02 21:23:38.334642 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.334614 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-openshift-ca" pod "d1f4bc4a-625a-4876-b92a-5b820b6d3041" (UID: "d1f4bc4a-625a-4876-b92a-5b820b6d3041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:23:38.336337 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.336311 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-xks-ca" pod "d1f4bc4a-625a-4876-b92a-5b820b6d3041" (UID: "d1f4bc4a-625a-4876-b92a-5b820b6d3041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:23:38.336472 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.336342 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f4bc4a-625a-4876-b92a-5b820b6d3041-kube-api-access-nltr8" pod "d1f4bc4a-625a-4876-b92a-5b820b6d3041" (UID: "d1f4bc4a-625a-4876-b92a-5b820b6d3041"). InnerVolumeSpecName "kube-api-access-nltr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:23:38.435831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.435745 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:38.435831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.435781 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1f4bc4a-625a-4876-b92a-5b820b6d3041-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:38.435831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:38.435790 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltr8\" (UniqueName: \"kubernetes.io/projected/d1f4bc4a-625a-4876-b92a-5b820b6d3041-kube-api-access-nltr8\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:39.109487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:39.109457 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" Nov 02 21:23:39.109487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:39.109471 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8" event={"ID":"d1f4bc4a-625a-4876-b92a-5b820b6d3041","Type":"ContainerDied","Data":"6b6aacfff8f8a49499218094eac939cb60f6e47b48df4428135045ede7aeb6ad"} Nov 02 21:23:39.109715 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:39.109521 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b6aacfff8f8a49499218094eac939cb60f6e47b48df4428135045ede7aeb6ad" Nov 02 21:23:51.686441 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:51.686396 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:23:51.688296 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:51.688248 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:23:52.695839 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:23:52.695802 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:23:55.015494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.015458 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l"] Nov 02 21:23:55.015905 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.015891 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d1f4bc4a-625a-4876-b92a-5b820b6d3041" containerName="revoke-keys" Nov 02 21:23:55.015957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.015906 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f4bc4a-625a-4876-b92a-5b820b6d3041" containerName="revoke-keys" Nov 02 21:23:55.016006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.015997 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f4bc4a-625a-4876-b92a-5b820b6d3041" containerName="revoke-keys" Nov 02 21:23:55.019487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.019469 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.045989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.045957 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305be545-6ccd-43b5-ab99-226e70fe44c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.046190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.046161 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlpr\" (UniqueName: \"kubernetes.io/projected/305be545-6ccd-43b5-ab99-226e70fe44c4-kube-api-access-qrlpr\") pod \"payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.121332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.121301 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fe61b17a-dockercfg-498tq\"" Nov 02 21:23:55.121332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.121271 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-742833\"" Nov 02 21:23:55.125405 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.125368 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l"] Nov 02 21:23:55.155895 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.148563 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlpr\" (UniqueName: \"kubernetes.io/projected/305be545-6ccd-43b5-ab99-226e70fe44c4-kube-api-access-qrlpr\") pod \"payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.155895 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.148668 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305be545-6ccd-43b5-ab99-226e70fe44c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.155895 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.149955 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305be545-6ccd-43b5-ab99-226e70fe44c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.177380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.177332 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9"] Nov 02 21:23:55.181351 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.181326 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.207708 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.207667 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlpr\" (UniqueName: \"kubernetes.io/projected/305be545-6ccd-43b5-ab99-226e70fe44c4-kube-api-access-qrlpr\") pod \"payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.233342 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.233308 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9"] Nov 02 21:23:55.249472 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.249433 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.249665 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.249576 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgpjx\" (UniqueName: \"kubernetes.io/projected/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-kube-api-access-mgpjx\") pod \"payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.332352 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.332244 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:55.351108 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.351055 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgpjx\" (UniqueName: \"kubernetes.io/projected/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-kube-api-access-mgpjx\") pod \"payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.351737 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.351198 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.352190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.352167 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.408047 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.408014 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgpjx\" (UniqueName: \"kubernetes.io/projected/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-kube-api-access-mgpjx\") pod \"payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.497485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.497449 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:55.701260 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.701228 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l"] Nov 02 21:23:55.702310 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:23:55.702252 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305be545_6ccd_43b5_ab99_226e70fe44c4.slice/crio-2162ff75b38197b969f499a20d046cfb66046a284026003059dbfb627bdee857 WatchSource:0}: Error finding container 2162ff75b38197b969f499a20d046cfb66046a284026003059dbfb627bdee857: Status 404 returned error can't find the container with id 2162ff75b38197b969f499a20d046cfb66046a284026003059dbfb627bdee857 Nov 02 21:23:55.863124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:55.863100 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9"] Nov 02 21:23:55.865483 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:23:55.865457 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1e7b4b_f721_4e48_b3cb_37f4e4f59617.slice/crio-0358f40a1b1fdf98180b4e3b36146b07672b508c6e030fdc8f11e8fb3d5879bb WatchSource:0}: Error finding container 0358f40a1b1fdf98180b4e3b36146b07672b508c6e030fdc8f11e8fb3d5879bb: Status 404 returned error can't find the container with id 0358f40a1b1fdf98180b4e3b36146b07672b508c6e030fdc8f11e8fb3d5879bb Nov 02 21:23:56.229475 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.229434 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" event={"ID":"305be545-6ccd-43b5-ab99-226e70fe44c4","Type":"ContainerStarted","Data":"7b3a291aeb37f9f687f6431ab1e16683e5d48f4a722fb16110e0bb0c5107f414"} Nov 02 21:23:56.229940 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.229513 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" event={"ID":"305be545-6ccd-43b5-ab99-226e70fe44c4","Type":"ContainerStarted","Data":"2162ff75b38197b969f499a20d046cfb66046a284026003059dbfb627bdee857"} Nov 02 21:23:56.229940 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.229655 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:23:56.231521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.231495 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" event={"ID":"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617","Type":"ContainerStarted","Data":"7b37174aec3ff9bf8befbd7254464400bf291a130ac8a995333a956ea7ee568d"} Nov 02 21:23:56.231724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.231540 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" event={"ID":"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617","Type":"ContainerStarted","Data":"0358f40a1b1fdf98180b4e3b36146b07672b508c6e030fdc8f11e8fb3d5879bb"} Nov 02 21:23:56.231724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.231638 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:23:56.338274 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.338221 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" podStartSLOduration=2.3382064590000002 podStartE2EDuration="2.338206459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:56.335678097 +0000 UTC m=+5734.852450681" watchObservedRunningTime="2026-11-02 21:23:56.338206459 +0000 UTC m=+5734.854979044" Nov 02 21:23:56.442314 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:23:56.442243 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" podStartSLOduration=1.442228099 podStartE2EDuration="1.442228099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:56.440570966 +0000 UTC m=+5734.957343552" watchObservedRunningTime="2026-11-02 21:23:56.442228099 +0000 UTC m=+5734.959000721" Nov 02 21:24:07.240404 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:24:07.240372 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:24:07.240947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:24:07.240441 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:24:22.696901 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:24:22.696862 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:24:26.879836 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:24:26.879807 5800 scope.go:122] "RemoveContainer" containerID="30a6c2e9ab36b70cf0dd8133cbde9a4bc74435001694c6d4c580ec9daf34220f" Nov 02 21:24:26.889339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:24:26.889315 5800 scope.go:122] "RemoveContainer" containerID="3493ee1d0b1eabaf2e2df53fdafe98667c5d99d3b901b97823efea0da0a74c71" Nov 02 21:24:52.696047 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:24:52.696005 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:25:04.644041 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.644003 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg"] Nov 02 21:25:04.648031 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.648009 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.697111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.697067 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gbfj\" (UniqueName: \"kubernetes.io/projected/7d501953-ffb9-46d9-95f4-8a09b5c64864-kube-api-access-4gbfj\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.697508 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.697127 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.697508 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.697212 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.744450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.744419 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:25:04.744631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.744489 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:25:04.744631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.744513 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:25:04.744631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.744532 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:25:04.747127 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.747108 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg"] Nov 02 21:25:04.797960 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.797926 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gbfj\" (UniqueName: \"kubernetes.io/projected/7d501953-ffb9-46d9-95f4-8a09b5c64864-kube-api-access-4gbfj\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.797960 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.797965 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.798196 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.797993 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.798889 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.798855 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.800862 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.800830 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.851160 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.851130 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gbfj\" (UniqueName: \"kubernetes.io/projected/7d501953-ffb9-46d9-95f4-8a09b5c64864-kube-api-access-4gbfj\") pod \"maas-api-revoke-sub-c427ac45-wb8zg\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:04.960998 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:04.960909 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:05.326138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:05.324128 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg"] Nov 02 21:25:05.327305 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:25:05.326916 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d501953_ffb9_46d9_95f4_8a09b5c64864.slice/crio-95b97047f991d49aab1a982cff3e634c4e0ee433399a128d1c7f210388c6fa1d WatchSource:0}: Error finding container 95b97047f991d49aab1a982cff3e634c4e0ee433399a128d1c7f210388c6fa1d: Status 404 returned error can't find the container with id 95b97047f991d49aab1a982cff3e634c4e0ee433399a128d1c7f210388c6fa1d Nov 02 21:25:05.636712 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:05.636685 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" event={"ID":"7d501953-ffb9-46d9-95f4-8a09b5c64864","Type":"ContainerStarted","Data":"95b97047f991d49aab1a982cff3e634c4e0ee433399a128d1c7f210388c6fa1d"} Nov 02 21:25:06.644169 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:06.644069 5800 generic.go:356] "Generic (PLEG): container finished" podID="7d501953-ffb9-46d9-95f4-8a09b5c64864" containerID="6233a88d12a754a41dcef1a2c5b7e03eb6392589d60b4b79ff8ca1df575b2a00" exitCode=0 Nov 02 21:25:06.644650 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:06.644165 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" event={"ID":"7d501953-ffb9-46d9-95f4-8a09b5c64864","Type":"ContainerDied","Data":"6233a88d12a754a41dcef1a2c5b7e03eb6392589d60b4b79ff8ca1df575b2a00"} Nov 02 21:25:07.790698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.790665 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:07.830356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.830324 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d501953-ffb9-46d9-95f4-8a09b5c64864-kube-api-access-4gbfj\" (UniqueName: \"kubernetes.io/projected/7d501953-ffb9-46d9-95f4-8a09b5c64864-kube-api-access-4gbfj\") pod \"7d501953-ffb9-46d9-95f4-8a09b5c64864\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " Nov 02 21:25:07.830556 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.830410 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-openshift-ca\") pod \"7d501953-ffb9-46d9-95f4-8a09b5c64864\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " Nov 02 21:25:07.830556 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.830527 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-xks-ca\") pod \"7d501953-ffb9-46d9-95f4-8a09b5c64864\" (UID: \"7d501953-ffb9-46d9-95f4-8a09b5c64864\") " Nov 02 21:25:07.830826 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.830791 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-openshift-ca" pod "7d501953-ffb9-46d9-95f4-8a09b5c64864" (UID: "7d501953-ffb9-46d9-95f4-8a09b5c64864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:25:07.832546 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.832512 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d501953-ffb9-46d9-95f4-8a09b5c64864-kube-api-access-4gbfj" pod "7d501953-ffb9-46d9-95f4-8a09b5c64864" (UID: "7d501953-ffb9-46d9-95f4-8a09b5c64864"). InnerVolumeSpecName "kube-api-access-4gbfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:25:07.832546 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.832518 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-xks-ca" pod "7d501953-ffb9-46d9-95f4-8a09b5c64864" (UID: "7d501953-ffb9-46d9-95f4-8a09b5c64864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:25:07.932390 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.932293 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:07.932390 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.932325 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d501953-ffb9-46d9-95f4-8a09b5c64864-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:07.932390 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:07.932337 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gbfj\" (UniqueName: \"kubernetes.io/projected/7d501953-ffb9-46d9-95f4-8a09b5c64864-kube-api-access-4gbfj\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:08.657025 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:08.656997 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" Nov 02 21:25:08.657025 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:08.657021 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg" event={"ID":"7d501953-ffb9-46d9-95f4-8a09b5c64864","Type":"ContainerDied","Data":"95b97047f991d49aab1a982cff3e634c4e0ee433399a128d1c7f210388c6fa1d"} Nov 02 21:25:08.657240 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:08.657061 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b97047f991d49aab1a982cff3e634c4e0ee433399a128d1c7f210388c6fa1d" Nov 02 21:25:22.661072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:22.661029 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9"] Nov 02 21:25:22.696703 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:25:22.696662 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:25:22.759121 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:22.759083 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l"] Nov 02 21:25:26.919587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:26.919554 5800 scope.go:122] "RemoveContainer" containerID="b0b21a860d9529bf9cf84521fd969fb6f8030059f9b51c39a6d9cb4982a39969" Nov 02 21:25:27.661435 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:27.661353 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" podUID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" containerName="payload-processing" containerID="cri-o://7b37174aec3ff9bf8befbd7254464400bf291a130ac8a995333a956ea7ee568d" gracePeriod=25 Nov 02 21:25:27.760225 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:27.760182 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" podUID="305be545-6ccd-43b5-ab99-226e70fe44c4" containerName="payload-pre-processing" containerID="cri-o://7b3a291aeb37f9f687f6431ab1e16683e5d48f4a722fb16110e0bb0c5107f414" gracePeriod=25 Nov 02 21:25:37.239207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:37.239163 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" podUID="305be545-6ccd-43b5-ab99-226e70fe44c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:25:37.239662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:37.239163 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" podUID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:25:47.239023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:47.238927 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" podUID="305be545-6ccd-43b5-ab99-226e70fe44c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:25:47.239559 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:47.238946 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" podUID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:25:47.884792 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:47.884758 5800 generic.go:356] "Generic (PLEG): container finished" podID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" containerID="7b37174aec3ff9bf8befbd7254464400bf291a130ac8a995333a956ea7ee568d" exitCode=0 Nov 02 21:25:47.884978 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:47.884827 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" event={"ID":"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617","Type":"ContainerDied","Data":"7b37174aec3ff9bf8befbd7254464400bf291a130ac8a995333a956ea7ee568d"} Nov 02 21:25:47.886745 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:47.886720 5800 generic.go:356] "Generic (PLEG): container finished" podID="305be545-6ccd-43b5-ab99-226e70fe44c4" containerID="7b3a291aeb37f9f687f6431ab1e16683e5d48f4a722fb16110e0bb0c5107f414" exitCode=0 Nov 02 21:25:47.886868 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:47.886770 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" event={"ID":"305be545-6ccd-43b5-ab99-226e70fe44c4","Type":"ContainerDied","Data":"7b3a291aeb37f9f687f6431ab1e16683e5d48f4a722fb16110e0bb0c5107f414"} Nov 02 21:25:47.925310 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:47.925228 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:25:48.010003 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.009977 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:25:48.064686 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.064652 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-plugins-config-volume\") pod \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " Nov 02 21:25:48.064869 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.064749 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305be545-6ccd-43b5-ab99-226e70fe44c4-kube-api-access-qrlpr\" (UniqueName: \"kubernetes.io/projected/305be545-6ccd-43b5-ab99-226e70fe44c4-kube-api-access-qrlpr\") pod \"305be545-6ccd-43b5-ab99-226e70fe44c4\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " Nov 02 21:25:48.064869 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.064791 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-kube-api-access-mgpjx\" (UniqueName: \"kubernetes.io/projected/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-kube-api-access-mgpjx\") pod \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\" (UID: \"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617\") " Nov 02 21:25:48.064869 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.064823 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305be545-6ccd-43b5-ab99-226e70fe44c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305be545-6ccd-43b5-ab99-226e70fe44c4-plugins-config-volume\") pod \"305be545-6ccd-43b5-ab99-226e70fe44c4\" (UID: \"305be545-6ccd-43b5-ab99-226e70fe44c4\") " Nov 02 21:25:48.065166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.065140 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-plugins-config-volume" pod "2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" (UID: "2a1e7b4b-f721-4e48-b3cb-37f4e4f59617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:25:48.065295 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.065255 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:48.065422 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.065325 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305be545-6ccd-43b5-ab99-226e70fe44c4-plugins-config-volume" pod "305be545-6ccd-43b5-ab99-226e70fe44c4" (UID: "305be545-6ccd-43b5-ab99-226e70fe44c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:25:48.067076 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.067044 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-kube-api-access-mgpjx" pod "2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" (UID: "2a1e7b4b-f721-4e48-b3cb-37f4e4f59617"). InnerVolumeSpecName "kube-api-access-mgpjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:25:48.067156 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.067127 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305be545-6ccd-43b5-ab99-226e70fe44c4-kube-api-access-qrlpr" pod "305be545-6ccd-43b5-ab99-226e70fe44c4" (UID: "305be545-6ccd-43b5-ab99-226e70fe44c4"). InnerVolumeSpecName "kube-api-access-qrlpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:25:48.166758 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.166718 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrlpr\" (UniqueName: \"kubernetes.io/projected/305be545-6ccd-43b5-ab99-226e70fe44c4-kube-api-access-qrlpr\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:48.166758 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.166753 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgpjx\" (UniqueName: \"kubernetes.io/projected/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617-kube-api-access-mgpjx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:48.166758 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.166764 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305be545-6ccd-43b5-ab99-226e70fe44c4-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:25:48.894908 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.894872 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" Nov 02 21:25:48.895417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.894909 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9" event={"ID":"2a1e7b4b-f721-4e48-b3cb-37f4e4f59617","Type":"ContainerDied","Data":"0358f40a1b1fdf98180b4e3b36146b07672b508c6e030fdc8f11e8fb3d5879bb"} Nov 02 21:25:48.895417 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.894969 5800 scope.go:122] "RemoveContainer" containerID="7b37174aec3ff9bf8befbd7254464400bf291a130ac8a995333a956ea7ee568d" Nov 02 21:25:48.896435 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.896350 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" event={"ID":"305be545-6ccd-43b5-ab99-226e70fe44c4","Type":"ContainerDied","Data":"2162ff75b38197b969f499a20d046cfb66046a284026003059dbfb627bdee857"} Nov 02 21:25:48.896435 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.896366 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l" Nov 02 21:25:48.904587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:48.904555 5800 scope.go:122] "RemoveContainer" containerID="7b3a291aeb37f9f687f6431ab1e16683e5d48f4a722fb16110e0bb0c5107f414" Nov 02 21:25:49.047754 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:49.047717 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l"] Nov 02 21:25:49.094842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:49.094807 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-742833-555bfh2b5l"] Nov 02 21:25:49.196340 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:49.196269 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9"] Nov 02 21:25:49.199747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:49.199719 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-742833-584f8d78clgsx9"] Nov 02 21:25:49.621470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:49.621438 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" path="/var/lib/kubelet/pods/2a1e7b4b-f721-4e48-b3cb-37f4e4f59617/volumes" Nov 02 21:25:49.621847 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:25:49.621834 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305be545-6ccd-43b5-ab99-226e70fe44c4" path="/var/lib/kubelet/pods/305be545-6ccd-43b5-ab99-226e70fe44c4/volumes" Nov 02 21:25:52.695235 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:25:52.695199 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:26:22.697925 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:26:22.697827 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:26:35.044106 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:35.044070 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8"] Nov 02 21:26:35.090251 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:35.090216 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qmmq8"] Nov 02 21:26:35.620756 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:35.620716 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca583944-4a9a-4a82-8034-ebf3bbdb045b" path="/var/lib/kubelet/pods/ca583944-4a9a-4a82-8034-ebf3bbdb045b/volumes" Nov 02 21:26:52.719311 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:26:52.719257 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:26:53.329965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:53.329403 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:26:53.329965 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:53.329506 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:26:54.041948 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:54.041910 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv"] Nov 02 21:26:54.087963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:54.087932 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-6v4mv"] Nov 02 21:26:55.628798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:26:55.628759 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3351ec-bd1e-4d7d-8412-62671a9991f7" path="/var/lib/kubelet/pods/7d3351ec-bd1e-4d7d-8412-62671a9991f7/volumes" Nov 02 21:27:05.041944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:05.041894 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw"] Nov 02 21:27:05.088095 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:05.088059 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-qqpkw"] Nov 02 21:27:05.621068 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:05.621031 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f2074c-c7f4-4d71-bd27-f1023234c93e" path="/var/lib/kubelet/pods/c8f2074c-c7f4-4d71-bd27-f1023234c93e/volumes" Nov 02 21:27:19.713506 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.713466 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw"] Nov 02 21:27:19.714033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.713961 5800 cpu_manager.go:414] "containerMap: removing container" podUID="305be545-6ccd-43b5-ab99-226e70fe44c4" containerName="payload-pre-processing" Nov 02 21:27:19.714033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.713973 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305be545-6ccd-43b5-ab99-226e70fe44c4" containerName="payload-pre-processing" Nov 02 21:27:19.714033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.713985 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" containerName="payload-processing" Nov 02 21:27:19.714033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.713991 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" containerName="payload-processing" Nov 02 21:27:19.714033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.714000 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7d501953-ffb9-46d9-95f4-8a09b5c64864" containerName="revoke-keys" Nov 02 21:27:19.714033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.714005 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d501953-ffb9-46d9-95f4-8a09b5c64864" containerName="revoke-keys" Nov 02 21:27:19.714275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.714099 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="305be545-6ccd-43b5-ab99-226e70fe44c4" containerName="payload-pre-processing" Nov 02 21:27:19.714275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.714108 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1e7b4b-f721-4e48-b3cb-37f4e4f59617" containerName="payload-processing" Nov 02 21:27:19.714275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.714116 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d501953-ffb9-46d9-95f4-8a09b5c64864" containerName="revoke-keys" Nov 02 21:27:19.717478 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.717460 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.764492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.764457 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.764708 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.764509 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.764708 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.764684 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxhc\" (UniqueName: \"kubernetes.io/projected/63b81348-1e20-45b8-b7b8-e830b4345cec-kube-api-access-fmxhc\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.813604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.813562 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:27:19.813819 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.813635 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:27:19.813819 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.813685 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:27:19.813819 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.813754 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:27:19.817306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.817263 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw"] Nov 02 21:27:19.865924 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.865879 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.866128 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.865931 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.866128 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.865997 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxhc\" (UniqueName: \"kubernetes.io/projected/63b81348-1e20-45b8-b7b8-e830b4345cec-kube-api-access-fmxhc\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.866833 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.866808 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.868496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.868470 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:19.919146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:19.919112 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxhc\" (UniqueName: \"kubernetes.io/projected/63b81348-1e20-45b8-b7b8-e830b4345cec-kube-api-access-fmxhc\") pod \"maas-api-revoke-sub-452b6d7b-b8gzw\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:20.032356 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:20.032316 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:20.401061 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:20.401029 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw"] Nov 02 21:27:20.403367 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:27:20.403339 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b81348_1e20_45b8_b7b8_e830b4345cec.slice/crio-76038e90e8eda9b0fc133914bc3c747c8cab534b10ceda850c5feb95ace75669 WatchSource:0}: Error finding container 76038e90e8eda9b0fc133914bc3c747c8cab534b10ceda850c5feb95ace75669: Status 404 returned error can't find the container with id 76038e90e8eda9b0fc133914bc3c747c8cab534b10ceda850c5feb95ace75669 Nov 02 21:27:20.448166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:20.448123 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" event={"ID":"63b81348-1e20-45b8-b7b8-e830b4345cec","Type":"ContainerStarted","Data":"76038e90e8eda9b0fc133914bc3c747c8cab534b10ceda850c5feb95ace75669"} Nov 02 21:27:21.460378 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:21.460275 5800 generic.go:356] "Generic (PLEG): container finished" podID="63b81348-1e20-45b8-b7b8-e830b4345cec" containerID="d9c9bd5cc65f3141398b0799b7c619d2457707faa48dbccaacf98d9ae883017a" exitCode=0 Nov 02 21:27:21.465949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:21.460684 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" event={"ID":"63b81348-1e20-45b8-b7b8-e830b4345cec","Type":"ContainerDied","Data":"d9c9bd5cc65f3141398b0799b7c619d2457707faa48dbccaacf98d9ae883017a"} Nov 02 21:27:22.823510 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:27:22.823465 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:27:22.915065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:22.915036 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:23.024976 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.024938 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-xks-ca\") pod \"63b81348-1e20-45b8-b7b8-e830b4345cec\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " Nov 02 21:27:23.025174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.024989 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-openshift-ca\") pod \"63b81348-1e20-45b8-b7b8-e830b4345cec\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " Nov 02 21:27:23.025174 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.025036 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b81348-1e20-45b8-b7b8-e830b4345cec-kube-api-access-fmxhc\" (UniqueName: \"kubernetes.io/projected/63b81348-1e20-45b8-b7b8-e830b4345cec-kube-api-access-fmxhc\") pod \"63b81348-1e20-45b8-b7b8-e830b4345cec\" (UID: \"63b81348-1e20-45b8-b7b8-e830b4345cec\") " Nov 02 21:27:23.025474 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.025443 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-openshift-ca" pod "63b81348-1e20-45b8-b7b8-e830b4345cec" (UID: "63b81348-1e20-45b8-b7b8-e830b4345cec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:27:23.061313 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.059475 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b81348-1e20-45b8-b7b8-e830b4345cec-kube-api-access-fmxhc" pod "63b81348-1e20-45b8-b7b8-e830b4345cec" (UID: "63b81348-1e20-45b8-b7b8-e830b4345cec"). InnerVolumeSpecName "kube-api-access-fmxhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:27:23.124422 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.124249 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-xks-ca" pod "63b81348-1e20-45b8-b7b8-e830b4345cec" (UID: "63b81348-1e20-45b8-b7b8-e830b4345cec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:27:23.142555 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.142506 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:23.142854 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.142832 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b81348-1e20-45b8-b7b8-e830b4345cec-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:23.142997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.142982 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxhc\" (UniqueName: \"kubernetes.io/projected/63b81348-1e20-45b8-b7b8-e830b4345cec-kube-api-access-fmxhc\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:23.507318 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.505434 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" event={"ID":"63b81348-1e20-45b8-b7b8-e830b4345cec","Type":"ContainerDied","Data":"76038e90e8eda9b0fc133914bc3c747c8cab534b10ceda850c5feb95ace75669"} Nov 02 21:27:23.507318 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.505502 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76038e90e8eda9b0fc133914bc3c747c8cab534b10ceda850c5feb95ace75669" Nov 02 21:27:23.507318 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.505577 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw" Nov 02 21:27:23.807536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.807439 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:27:23.807536 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:23.807501 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:27:38.794363 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:38.793399 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:27:38.794363 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:38.793478 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:27:40.409371 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.409328 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs"] Nov 02 21:27:40.410214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.410188 5800 cpu_manager.go:414] "containerMap: removing container" podUID="63b81348-1e20-45b8-b7b8-e830b4345cec" containerName="revoke-keys" Nov 02 21:27:40.410214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.410212 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b81348-1e20-45b8-b7b8-e830b4345cec" containerName="revoke-keys" Nov 02 21:27:40.410438 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.410380 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b81348-1e20-45b8-b7b8-e830b4345cec" containerName="revoke-keys" Nov 02 21:27:40.413423 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.413401 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.462964 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.462922 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:27:40.463172 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.463137 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:27:40.466307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.466262 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs"] Nov 02 21:27:40.509431 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.509394 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:27:40.509431 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.509411 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:27:40.510973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.510949 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.511087 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.510980 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w97pq\" (UniqueName: \"kubernetes.io/projected/60d292af-6d43-4dd1-9525-5d478fb7e15e-kube-api-access-w97pq\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.511134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.511086 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.611782 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.611748 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.611782 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.611784 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w97pq\" (UniqueName: \"kubernetes.io/projected/60d292af-6d43-4dd1-9525-5d478fb7e15e-kube-api-access-w97pq\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.612067 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.611841 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.612552 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.612522 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.614384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.614360 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.665027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.664948 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w97pq\" (UniqueName: \"kubernetes.io/projected/60d292af-6d43-4dd1-9525-5d478fb7e15e-kube-api-access-w97pq\") pod \"maas-api-revoke-sub-3e31a506-t4txs\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:40.731416 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:40.731381 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:41.097624 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:41.097591 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs"] Nov 02 21:27:41.099254 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:27:41.099229 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d292af_6d43_4dd1_9525_5d478fb7e15e.slice/crio-efdcbe76167e1d812bf77f2fad5001ff298657a37375cada0c104b3cf9385d91 WatchSource:0}: Error finding container efdcbe76167e1d812bf77f2fad5001ff298657a37375cada0c104b3cf9385d91: Status 404 returned error can't find the container with id efdcbe76167e1d812bf77f2fad5001ff298657a37375cada0c104b3cf9385d91 Nov 02 21:27:41.628412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:41.628368 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" event={"ID":"60d292af-6d43-4dd1-9525-5d478fb7e15e","Type":"ContainerStarted","Data":"8291f7eb8024ba2ac054022484c5ed21eeb3589c1b37ef35ae32d690a5ac422f"} Nov 02 21:27:41.628854 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:41.628436 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" event={"ID":"60d292af-6d43-4dd1-9525-5d478fb7e15e","Type":"ContainerStarted","Data":"efdcbe76167e1d812bf77f2fad5001ff298657a37375cada0c104b3cf9385d91"} Nov 02 21:27:41.736781 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:41.736669 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" podStartSLOduration=1.736654407 podStartE2EDuration="1.736654407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:27:41.734518318 +0000 UTC m=+5960.251290903" watchObservedRunningTime="2026-11-02 21:27:41.736654407 +0000 UTC m=+5960.253426991" Nov 02 21:27:42.635835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:42.635797 5800 generic.go:356] "Generic (PLEG): container finished" podID="60d292af-6d43-4dd1-9525-5d478fb7e15e" containerID="8291f7eb8024ba2ac054022484c5ed21eeb3589c1b37ef35ae32d690a5ac422f" exitCode=0 Nov 02 21:27:42.636300 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:42.635887 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" event={"ID":"60d292af-6d43-4dd1-9525-5d478fb7e15e","Type":"ContainerDied","Data":"8291f7eb8024ba2ac054022484c5ed21eeb3589c1b37ef35ae32d690a5ac422f"} Nov 02 21:27:43.913657 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:43.913630 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:43.999308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:43.997577 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d292af-6d43-4dd1-9525-5d478fb7e15e-kube-api-access-w97pq\" (UniqueName: \"kubernetes.io/projected/60d292af-6d43-4dd1-9525-5d478fb7e15e-kube-api-access-w97pq\") pod \"60d292af-6d43-4dd1-9525-5d478fb7e15e\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " Nov 02 21:27:43.999308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:43.997679 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-openshift-ca\") pod \"60d292af-6d43-4dd1-9525-5d478fb7e15e\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " Nov 02 21:27:43.999308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:43.997774 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-xks-ca\") pod \"60d292af-6d43-4dd1-9525-5d478fb7e15e\" (UID: \"60d292af-6d43-4dd1-9525-5d478fb7e15e\") " Nov 02 21:27:44.000257 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.000221 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-openshift-ca" pod "60d292af-6d43-4dd1-9525-5d478fb7e15e" (UID: "60d292af-6d43-4dd1-9525-5d478fb7e15e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:27:44.002812 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.002765 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-xks-ca" pod "60d292af-6d43-4dd1-9525-5d478fb7e15e" (UID: "60d292af-6d43-4dd1-9525-5d478fb7e15e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:27:44.003825 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.003784 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d292af-6d43-4dd1-9525-5d478fb7e15e-kube-api-access-w97pq" pod "60d292af-6d43-4dd1-9525-5d478fb7e15e" (UID: "60d292af-6d43-4dd1-9525-5d478fb7e15e"). InnerVolumeSpecName "kube-api-access-w97pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:27:44.104017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.103903 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w97pq\" (UniqueName: \"kubernetes.io/projected/60d292af-6d43-4dd1-9525-5d478fb7e15e-kube-api-access-w97pq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:44.104017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.103947 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:44.104017 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.103967 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60d292af-6d43-4dd1-9525-5d478fb7e15e-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:27:44.673385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.663746 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" event={"ID":"60d292af-6d43-4dd1-9525-5d478fb7e15e","Type":"ContainerDied","Data":"efdcbe76167e1d812bf77f2fad5001ff298657a37375cada0c104b3cf9385d91"} Nov 02 21:27:44.673385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.663814 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efdcbe76167e1d812bf77f2fad5001ff298657a37375cada0c104b3cf9385d91" Nov 02 21:27:44.673385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:27:44.663886 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs" Nov 02 21:27:52.699326 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:27:52.699265 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:28:02.145553 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.145512 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8"] Nov 02 21:28:02.146322 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.146298 5800 cpu_manager.go:414] "containerMap: removing container" podUID="60d292af-6d43-4dd1-9525-5d478fb7e15e" containerName="revoke-keys" Nov 02 21:28:02.146401 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.146323 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d292af-6d43-4dd1-9525-5d478fb7e15e" containerName="revoke-keys" Nov 02 21:28:02.146456 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.146445 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d292af-6d43-4dd1-9525-5d478fb7e15e" containerName="revoke-keys" Nov 02 21:28:02.150615 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.150590 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.201011 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.200971 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8"] Nov 02 21:28:02.209946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.209904 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6tv\" (UniqueName: \"kubernetes.io/projected/e1424fad-385b-4927-8378-0652243a53d3-kube-api-access-rk6tv\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.210114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.210009 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1424fad-385b-4927-8378-0652243a53d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.210114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.210070 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1424fad-385b-4927-8378-0652243a53d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.243470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.243426 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:28:02.243662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.243491 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:28:02.243662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.243591 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:28:02.245161 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.245132 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:28:02.311062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.311022 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1424fad-385b-4927-8378-0652243a53d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.311264 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.311075 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6tv\" (UniqueName: \"kubernetes.io/projected/e1424fad-385b-4927-8378-0652243a53d3-kube-api-access-rk6tv\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.311264 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.311145 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1424fad-385b-4927-8378-0652243a53d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.311751 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.311725 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1424fad-385b-4927-8378-0652243a53d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.313614 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.313588 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1424fad-385b-4927-8378-0652243a53d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.364328 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.364293 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6tv\" (UniqueName: \"kubernetes.io/projected/e1424fad-385b-4927-8378-0652243a53d3-kube-api-access-rk6tv\") pod \"maas-api-revoke-sub-941f5cfd-5prz8\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.464539 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.464443 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:02.841701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.841676 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8"] Nov 02 21:28:02.843929 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:28:02.843895 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1424fad_385b_4927_8378_0652243a53d3.slice/crio-c5c45465adb4a1c9f8092b744e7e219d0e2e6c2956b40e0857b8ddfbd574f7e8 WatchSource:0}: Error finding container c5c45465adb4a1c9f8092b744e7e219d0e2e6c2956b40e0857b8ddfbd574f7e8: Status 404 returned error can't find the container with id c5c45465adb4a1c9f8092b744e7e219d0e2e6c2956b40e0857b8ddfbd574f7e8 Nov 02 21:28:02.875523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:02.875489 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" event={"ID":"e1424fad-385b-4927-8378-0652243a53d3","Type":"ContainerStarted","Data":"c5c45465adb4a1c9f8092b744e7e219d0e2e6c2956b40e0857b8ddfbd574f7e8"} Nov 02 21:28:03.882268 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:03.882233 5800 generic.go:356] "Generic (PLEG): container finished" podID="e1424fad-385b-4927-8378-0652243a53d3" containerID="437f2bdd476154d87a36cfb800129c2cf7c2dcf1c1fac15e7ef5c8d120a53aee" exitCode=0 Nov 02 21:28:03.882671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:03.882323 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" event={"ID":"e1424fad-385b-4927-8378-0652243a53d3","Type":"ContainerDied","Data":"437f2bdd476154d87a36cfb800129c2cf7c2dcf1c1fac15e7ef5c8d120a53aee"} Nov 02 21:28:05.034178 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.034146 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:05.142016 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.141923 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1424fad-385b-4927-8378-0652243a53d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1424fad-385b-4927-8378-0652243a53d3-maas-api-openshift-ca\") pod \"e1424fad-385b-4927-8378-0652243a53d3\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " Nov 02 21:28:05.142220 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.142083 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1424fad-385b-4927-8378-0652243a53d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1424fad-385b-4927-8378-0652243a53d3-maas-api-xks-ca\") pod \"e1424fad-385b-4927-8378-0652243a53d3\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " Nov 02 21:28:05.142220 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.142124 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1424fad-385b-4927-8378-0652243a53d3-kube-api-access-rk6tv\" (UniqueName: \"kubernetes.io/projected/e1424fad-385b-4927-8378-0652243a53d3-kube-api-access-rk6tv\") pod \"e1424fad-385b-4927-8378-0652243a53d3\" (UID: \"e1424fad-385b-4927-8378-0652243a53d3\") " Nov 02 21:28:05.142496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.142465 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1424fad-385b-4927-8378-0652243a53d3-maas-api-openshift-ca" pod "e1424fad-385b-4927-8378-0652243a53d3" (UID: "e1424fad-385b-4927-8378-0652243a53d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:05.144879 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.144845 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1424fad-385b-4927-8378-0652243a53d3-kube-api-access-rk6tv" pod "e1424fad-385b-4927-8378-0652243a53d3" (UID: "e1424fad-385b-4927-8378-0652243a53d3"). InnerVolumeSpecName "kube-api-access-rk6tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:05.144996 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.144979 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1424fad-385b-4927-8378-0652243a53d3-maas-api-xks-ca" pod "e1424fad-385b-4927-8378-0652243a53d3" (UID: "e1424fad-385b-4927-8378-0652243a53d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:28:05.243190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.243147 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1424fad-385b-4927-8378-0652243a53d3-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:05.243190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.243183 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6tv\" (UniqueName: \"kubernetes.io/projected/e1424fad-385b-4927-8378-0652243a53d3-kube-api-access-rk6tv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:05.243190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.243195 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1424fad-385b-4927-8378-0652243a53d3-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:05.895547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.895457 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" Nov 02 21:28:05.895547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.895484 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8" event={"ID":"e1424fad-385b-4927-8378-0652243a53d3","Type":"ContainerDied","Data":"c5c45465adb4a1c9f8092b744e7e219d0e2e6c2956b40e0857b8ddfbd574f7e8"} Nov 02 21:28:05.895547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:05.895543 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c45465adb4a1c9f8092b744e7e219d0e2e6c2956b40e0857b8ddfbd574f7e8" Nov 02 21:28:20.042180 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:20.042132 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v"] Nov 02 21:28:20.088685 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:20.088643 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3035529-kzx8v"] Nov 02 21:28:21.630629 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:21.630592 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f50641-f653-4697-b325-d09cbedb5594" path="/var/lib/kubelet/pods/d2f50641-f653-4697-b325-d09cbedb5594/volumes" Nov 02 21:28:22.696202 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:28:22.696163 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:28:22.731977 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:22.731943 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:28:22.744473 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:22.744447 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:28:26.973446 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:26.973408 5800 scope.go:122] "RemoveContainer" containerID="9e2364fe3b4e7cdcea9ff032db88d11f1e21766bc05ac9496f4f6650dd6ed2b5" Nov 02 21:28:26.989403 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:26.989376 5800 scope.go:122] "RemoveContainer" containerID="9f32161a60e24437774dda8346e4feb13b6e90a731a323eb7432889ad33f5e60" Nov 02 21:28:26.998309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:26.998268 5800 scope.go:122] "RemoveContainer" containerID="4215162b4cebd4f77e877cee8c64655a194241763fd8c77d518d70f03e53c2c2" Nov 02 21:28:31.329199 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:31.328099 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42ksz"] Nov 02 21:28:36.329684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.329617 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" podUID="0cd9fdd9-fd6a-415a-9d89-655b5637a192" containerName="payload-pre-processing" containerID="cri-o://e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80" gracePeriod=25 Nov 02 21:28:36.582677 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.582610 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 21:28:36.764031 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.763988 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd9fdd9-fd6a-415a-9d89-655b5637a192-kube-api-access-gn979\" (UniqueName: \"kubernetes.io/projected/0cd9fdd9-fd6a-415a-9d89-655b5637a192-kube-api-access-gn979\") pod \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " Nov 02 21:28:36.764216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.764090 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd9fdd9-fd6a-415a-9d89-655b5637a192-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd9fdd9-fd6a-415a-9d89-655b5637a192-plugins-config-volume\") pod \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\" (UID: \"0cd9fdd9-fd6a-415a-9d89-655b5637a192\") " Nov 02 21:28:36.764488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.764455 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd9fdd9-fd6a-415a-9d89-655b5637a192-plugins-config-volume" pod "0cd9fdd9-fd6a-415a-9d89-655b5637a192" (UID: "0cd9fdd9-fd6a-415a-9d89-655b5637a192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:28:36.764730 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.764706 5800 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd9fdd9-fd6a-415a-9d89-655b5637a192-plugins-config-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:36.766326 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.766306 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd9fdd9-fd6a-415a-9d89-655b5637a192-kube-api-access-gn979" pod "0cd9fdd9-fd6a-415a-9d89-655b5637a192" (UID: "0cd9fdd9-fd6a-415a-9d89-655b5637a192"). InnerVolumeSpecName "kube-api-access-gn979". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:28:36.865998 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:36.865909 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn979\" (UniqueName: \"kubernetes.io/projected/0cd9fdd9-fd6a-415a-9d89-655b5637a192-kube-api-access-gn979\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:28:37.082620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.082581 5800 generic.go:356] "Generic (PLEG): container finished" podID="0cd9fdd9-fd6a-415a-9d89-655b5637a192" containerID="e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80" exitCode=0 Nov 02 21:28:37.082802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.082626 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" Nov 02 21:28:37.082802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.082670 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" event={"ID":"0cd9fdd9-fd6a-415a-9d89-655b5637a192","Type":"ContainerDied","Data":"e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80"} Nov 02 21:28:37.082802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.082726 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-42ksz" event={"ID":"0cd9fdd9-fd6a-415a-9d89-655b5637a192","Type":"ContainerDied","Data":"6a65478cafc38553b344e21dd70113ce277574e11bc0c0008569736640733c62"} Nov 02 21:28:37.082802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.082752 5800 scope.go:122] "RemoveContainer" containerID="e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80" Nov 02 21:28:37.093946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.093921 5800 scope.go:122] "RemoveContainer" containerID="e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80" Nov 02 21:28:37.094264 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:28:37.094244 5800 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80\": container with ID starting with e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80 not found: ID does not exist" containerID="e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80" Nov 02 21:28:37.094333 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.094274 5800 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80"} err="failed to get container status \"e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80\": rpc error: code = NotFound desc = could not find container \"e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80\": container with ID starting with e5656c211ba6108a5f44a5d619f17af380816e16f983641aec7e9e7e72e3bc80 not found: ID does not exist" Nov 02 21:28:37.235464 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.235384 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42ksz"] Nov 02 21:28:37.239311 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.239269 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-42ksz"] Nov 02 21:28:37.620481 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:37.620444 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd9fdd9-fd6a-415a-9d89-655b5637a192" path="/var/lib/kubelet/pods/0cd9fdd9-fd6a-415a-9d89-655b5637a192/volumes" Nov 02 21:28:39.048906 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:39.048869 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8"] Nov 02 21:28:39.095620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:39.095592 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05ad9e50-lrjm8"] Nov 02 21:28:39.620445 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:28:39.620409 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f4bc4a-625a-4876-b92a-5b820b6d3041" path="/var/lib/kubelet/pods/d1f4bc4a-625a-4876-b92a-5b820b6d3041/volumes" Nov 02 21:28:52.696969 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:28:52.696928 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:29:22.695482 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:29:22.695444 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:29:27.033214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:29:27.033171 5800 scope.go:122] "RemoveContainer" containerID="db7496c5c40d79f6d39524e278db372a0c4377f4e00a89bab3aa1c5cb9de8b48" Nov 02 21:29:52.695337 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:29:52.695296 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:30:00.158577 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.158536 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz"] Nov 02 21:30:00.159125 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.159109 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e1424fad-385b-4927-8378-0652243a53d3" containerName="revoke-keys" Nov 02 21:30:00.159175 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.159127 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1424fad-385b-4927-8378-0652243a53d3" containerName="revoke-keys" Nov 02 21:30:00.159175 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.159145 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0cd9fdd9-fd6a-415a-9d89-655b5637a192" containerName="payload-pre-processing" Nov 02 21:30:00.159175 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.159153 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd9fdd9-fd6a-415a-9d89-655b5637a192" containerName="payload-pre-processing" Nov 02 21:30:00.159327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.159314 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1424fad-385b-4927-8378-0652243a53d3" containerName="revoke-keys" Nov 02 21:30:00.159374 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.159334 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd9fdd9-fd6a-415a-9d89-655b5637a192" containerName="payload-pre-processing" Nov 02 21:30:00.163311 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.163266 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.214993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.214958 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz"] Nov 02 21:30:00.255882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.255829 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:30:00.256305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.256251 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:30:00.258166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.258140 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:30:00.318467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.318432 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918a1961-8e71-4eee-96ed-be60fd9dd8ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-mm7jz\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.318467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.318470 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6cm\" (UniqueName: \"kubernetes.io/projected/918a1961-8e71-4eee-96ed-be60fd9dd8ff-kube-api-access-qm6cm\") pod \"maas-api-key-cleanup-29894250-mm7jz\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.423748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.419918 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918a1961-8e71-4eee-96ed-be60fd9dd8ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-mm7jz\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.423748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.419972 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6cm\" (UniqueName: \"kubernetes.io/projected/918a1961-8e71-4eee-96ed-be60fd9dd8ff-kube-api-access-qm6cm\") pod \"maas-api-key-cleanup-29894250-mm7jz\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.423748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.421189 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918a1961-8e71-4eee-96ed-be60fd9dd8ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894250-mm7jz\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.473816 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.473784 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6cm\" (UniqueName: \"kubernetes.io/projected/918a1961-8e71-4eee-96ed-be60fd9dd8ff-kube-api-access-qm6cm\") pod \"maas-api-key-cleanup-29894250-mm7jz\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.480522 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.480496 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:00.847025 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:00.846991 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz"] Nov 02 21:30:00.848941 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:30:00.848917 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918a1961_8e71_4eee_96ed_be60fd9dd8ff.slice/crio-4422b172cd3a4b13f66fe94309a110b24c4d2fa3de3fd2f8c2eb4a0b05951fc7 WatchSource:0}: Error finding container 4422b172cd3a4b13f66fe94309a110b24c4d2fa3de3fd2f8c2eb4a0b05951fc7: Status 404 returned error can't find the container with id 4422b172cd3a4b13f66fe94309a110b24c4d2fa3de3fd2f8c2eb4a0b05951fc7 Nov 02 21:30:01.588631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:01.588593 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" event={"ID":"918a1961-8e71-4eee-96ed-be60fd9dd8ff","Type":"ContainerStarted","Data":"5e41a4425e1ec7eb101ea8d568b2d01961510a2b090961ea2eb5889bc24b4b92"} Nov 02 21:30:01.589021 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:01.588653 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" event={"ID":"918a1961-8e71-4eee-96ed-be60fd9dd8ff","Type":"ContainerStarted","Data":"4422b172cd3a4b13f66fe94309a110b24c4d2fa3de3fd2f8c2eb4a0b05951fc7"} Nov 02 21:30:01.694606 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:01.694553 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" podStartSLOduration=1.69454039 podStartE2EDuration="1.69454039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:01.69352934 +0000 UTC m=+6100.210301930" watchObservedRunningTime="2026-11-02 21:30:01.69454039 +0000 UTC m=+6100.211312972" Nov 02 21:30:02.596376 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:02.596341 5800 generic.go:356] "Generic (PLEG): container finished" podID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerID="5e41a4425e1ec7eb101ea8d568b2d01961510a2b090961ea2eb5889bc24b4b92" exitCode=60 Nov 02 21:30:02.596772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:02.596425 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" event={"ID":"918a1961-8e71-4eee-96ed-be60fd9dd8ff","Type":"ContainerDied","Data":"5e41a4425e1ec7eb101ea8d568b2d01961510a2b090961ea2eb5889bc24b4b92"} Nov 02 21:30:02.596772 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:02.596696 5800 scope.go:122] "RemoveContainer" containerID="5e41a4425e1ec7eb101ea8d568b2d01961510a2b090961ea2eb5889bc24b4b92" Nov 02 21:30:03.604783 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:03.604748 5800 generic.go:356] "Generic (PLEG): container finished" podID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerID="4c8275e1245841e2ebc546443dc1d009d937dca15658e71acaa76c3a0d3a978f" exitCode=60 Nov 02 21:30:03.605239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:03.604831 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" event={"ID":"918a1961-8e71-4eee-96ed-be60fd9dd8ff","Type":"ContainerDied","Data":"4c8275e1245841e2ebc546443dc1d009d937dca15658e71acaa76c3a0d3a978f"} Nov 02 21:30:03.605239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:03.604934 5800 scope.go:122] "RemoveContainer" containerID="5e41a4425e1ec7eb101ea8d568b2d01961510a2b090961ea2eb5889bc24b4b92" Nov 02 21:30:03.605239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:03.605195 5800 scope.go:122] "RemoveContainer" containerID="4c8275e1245841e2ebc546443dc1d009d937dca15658e71acaa76c3a0d3a978f" Nov 02 21:30:03.605498 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:30:03.605473 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-mm7jz_odh-ai-gateway-infra(918a1961-8e71-4eee-96ed-be60fd9dd8ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" Nov 02 21:30:04.612414 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:04.612386 5800 scope.go:122] "RemoveContainer" containerID="4c8275e1245841e2ebc546443dc1d009d937dca15658e71acaa76c3a0d3a978f" Nov 02 21:30:04.612809 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:30:04.612629 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-mm7jz_odh-ai-gateway-infra(918a1961-8e71-4eee-96ed-be60fd9dd8ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" Nov 02 21:30:08.043110 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:08.043074 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg"] Nov 02 21:30:08.089062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:08.089029 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c427ac45-wb8zg"] Nov 02 21:30:09.621804 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:09.621765 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d501953-ffb9-46d9-95f4-8a09b5c64864" path="/var/lib/kubelet/pods/7d501953-ffb9-46d9-95f4-8a09b5c64864/volumes" Nov 02 21:30:16.615033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:16.615002 5800 scope.go:122] "RemoveContainer" containerID="4c8275e1245841e2ebc546443dc1d009d937dca15658e71acaa76c3a0d3a978f" Nov 02 21:30:17.690644 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:17.690561 5800 generic.go:356] "Generic (PLEG): container finished" podID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerID="6513df310fb9ed1c4413f02e9c573199b1958f29e222c37c4219b8566d8d220d" exitCode=60 Nov 02 21:30:17.691112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:17.690636 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" event={"ID":"918a1961-8e71-4eee-96ed-be60fd9dd8ff","Type":"ContainerDied","Data":"6513df310fb9ed1c4413f02e9c573199b1958f29e222c37c4219b8566d8d220d"} Nov 02 21:30:17.691112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:17.690700 5800 scope.go:122] "RemoveContainer" containerID="4c8275e1245841e2ebc546443dc1d009d937dca15658e71acaa76c3a0d3a978f" Nov 02 21:30:17.691246 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:17.691126 5800 scope.go:122] "RemoveContainer" containerID="6513df310fb9ed1c4413f02e9c573199b1958f29e222c37c4219b8566d8d220d" Nov 02 21:30:17.691487 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:30:17.691446 5800 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894250-mm7jz_odh-ai-gateway-infra(918a1961-8e71-4eee-96ed-be60fd9dd8ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" Nov 02 21:30:18.806432 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:18.806391 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz"] Nov 02 21:30:18.953170 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:18.953137 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:19.023926 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.023893 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918a1961-8e71-4eee-96ed-be60fd9dd8ff-kube-api-access-qm6cm\" (UniqueName: \"kubernetes.io/projected/918a1961-8e71-4eee-96ed-be60fd9dd8ff-kube-api-access-qm6cm\") pod \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " Nov 02 21:30:19.024109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.023975 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918a1961-8e71-4eee-96ed-be60fd9dd8ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918a1961-8e71-4eee-96ed-be60fd9dd8ff-maas-api-service-ca\") pod \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\" (UID: \"918a1961-8e71-4eee-96ed-be60fd9dd8ff\") " Nov 02 21:30:19.024378 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.024356 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918a1961-8e71-4eee-96ed-be60fd9dd8ff-maas-api-service-ca" pod "918a1961-8e71-4eee-96ed-be60fd9dd8ff" (UID: "918a1961-8e71-4eee-96ed-be60fd9dd8ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:19.026038 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.026014 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918a1961-8e71-4eee-96ed-be60fd9dd8ff-kube-api-access-qm6cm" pod "918a1961-8e71-4eee-96ed-be60fd9dd8ff" (UID: "918a1961-8e71-4eee-96ed-be60fd9dd8ff"). InnerVolumeSpecName "kube-api-access-qm6cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:19.125347 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.125243 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/918a1961-8e71-4eee-96ed-be60fd9dd8ff-maas-api-service-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:19.125347 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.125323 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6cm\" (UniqueName: \"kubernetes.io/projected/918a1961-8e71-4eee-96ed-be60fd9dd8ff-kube-api-access-qm6cm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:19.705336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.705304 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" Nov 02 21:30:19.705336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.705318 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz" event={"ID":"918a1961-8e71-4eee-96ed-be60fd9dd8ff","Type":"ContainerDied","Data":"4422b172cd3a4b13f66fe94309a110b24c4d2fa3de3fd2f8c2eb4a0b05951fc7"} Nov 02 21:30:19.705617 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.705395 5800 scope.go:122] "RemoveContainer" containerID="6513df310fb9ed1c4413f02e9c573199b1958f29e222c37c4219b8566d8d220d" Nov 02 21:30:19.858250 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.858210 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz"] Nov 02 21:30:19.904229 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:19.904146 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894250-mm7jz"] Nov 02 21:30:21.621299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:21.621256 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" path="/var/lib/kubelet/pods/918a1961-8e71-4eee-96ed-be60fd9dd8ff/volumes" Nov 02 21:30:22.696762 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:30:22.696722 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:30:24.381096 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.381060 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4"] Nov 02 21:30:24.381533 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.381505 5800 cpu_manager.go:414] "containerMap: removing container" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:24.381533 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.381519 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:24.381613 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.381553 5800 cpu_manager.go:414] "containerMap: removing container" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:24.381613 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.381558 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:24.381679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.381637 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:24.381679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.381647 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:24.386913 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.386893 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.437474 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.437438 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4"] Nov 02 21:30:24.480333 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.480266 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.480583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.480343 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.480583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.480453 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgqs\" (UniqueName: \"kubernetes.io/projected/740b0973-6474-4ae0-b9bf-c55033a946c9-kube-api-access-wqgqs\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.480777 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.480761 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:30:24.480845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.480779 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:30:24.480845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.480803 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:30:24.481842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.481825 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:30:24.581211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.581180 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.581211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.581218 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.581526 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.581261 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgqs\" (UniqueName: \"kubernetes.io/projected/740b0973-6474-4ae0-b9bf-c55033a946c9-kube-api-access-wqgqs\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.581949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.581925 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.583671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.583650 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.634462 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.634392 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgqs\" (UniqueName: \"kubernetes.io/projected/740b0973-6474-4ae0-b9bf-c55033a946c9-kube-api-access-wqgqs\") pod \"maas-api-revoke-sub-e2547622-lmmd4\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:24.700391 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:24.700357 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:25.063901 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:25.063877 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4"] Nov 02 21:30:25.065919 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:30:25.065886 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740b0973_6474_4ae0_b9bf_c55033a946c9.slice/crio-fd842eae5943b90fcda246024796830eea809b913b01e6d1303818dc2e574e01 WatchSource:0}: Error finding container fd842eae5943b90fcda246024796830eea809b913b01e6d1303818dc2e574e01: Status 404 returned error can't find the container with id fd842eae5943b90fcda246024796830eea809b913b01e6d1303818dc2e574e01 Nov 02 21:30:25.740868 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:25.740776 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" event={"ID":"740b0973-6474-4ae0-b9bf-c55033a946c9","Type":"ContainerStarted","Data":"4912bbe925d203d2b7b6146e11c86a3f409b260f8ff625c9006143d4a7cea911"} Nov 02 21:30:25.740868 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:25.740831 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" event={"ID":"740b0973-6474-4ae0-b9bf-c55033a946c9","Type":"ContainerStarted","Data":"fd842eae5943b90fcda246024796830eea809b913b01e6d1303818dc2e574e01"} Nov 02 21:30:25.846951 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:25.846893 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" podStartSLOduration=1.846879184 podStartE2EDuration="1.846879184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:25.845851489 +0000 UTC m=+6124.362624076" watchObservedRunningTime="2026-11-02 21:30:25.846879184 +0000 UTC m=+6124.363651769" Nov 02 21:30:26.748271 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:26.748233 5800 generic.go:356] "Generic (PLEG): container finished" podID="740b0973-6474-4ae0-b9bf-c55033a946c9" containerID="4912bbe925d203d2b7b6146e11c86a3f409b260f8ff625c9006143d4a7cea911" exitCode=0 Nov 02 21:30:26.748729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:26.748320 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" event={"ID":"740b0973-6474-4ae0-b9bf-c55033a946c9","Type":"ContainerDied","Data":"4912bbe925d203d2b7b6146e11c86a3f409b260f8ff625c9006143d4a7cea911"} Nov 02 21:30:27.059843 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:27.059808 5800 scope.go:122] "RemoveContainer" containerID="429f7686307a1cea71c9f13efaa43d61a52ae4cd159c723804371fff8746f1ce" Nov 02 21:30:27.891248 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:27.891220 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:28.018952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.018863 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740b0973-6474-4ae0-b9bf-c55033a946c9-kube-api-access-wqgqs\" (UniqueName: \"kubernetes.io/projected/740b0973-6474-4ae0-b9bf-c55033a946c9-kube-api-access-wqgqs\") pod \"740b0973-6474-4ae0-b9bf-c55033a946c9\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " Nov 02 21:30:28.018952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.018934 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-openshift-ca\") pod \"740b0973-6474-4ae0-b9bf-c55033a946c9\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " Nov 02 21:30:28.019201 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.018967 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-xks-ca\") pod \"740b0973-6474-4ae0-b9bf-c55033a946c9\" (UID: \"740b0973-6474-4ae0-b9bf-c55033a946c9\") " Nov 02 21:30:28.019379 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.019347 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-openshift-ca" pod "740b0973-6474-4ae0-b9bf-c55033a946c9" (UID: "740b0973-6474-4ae0-b9bf-c55033a946c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:28.019487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.019444 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.021103 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.021072 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-xks-ca" pod "740b0973-6474-4ae0-b9bf-c55033a946c9" (UID: "740b0973-6474-4ae0-b9bf-c55033a946c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:30:28.021204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.021147 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740b0973-6474-4ae0-b9bf-c55033a946c9-kube-api-access-wqgqs" pod "740b0973-6474-4ae0-b9bf-c55033a946c9" (UID: "740b0973-6474-4ae0-b9bf-c55033a946c9"). InnerVolumeSpecName "kube-api-access-wqgqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:28.120507 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.120465 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgqs\" (UniqueName: \"kubernetes.io/projected/740b0973-6474-4ae0-b9bf-c55033a946c9-kube-api-access-wqgqs\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.120507 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.120502 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740b0973-6474-4ae0-b9bf-c55033a946c9-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.760109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.760077 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" event={"ID":"740b0973-6474-4ae0-b9bf-c55033a946c9","Type":"ContainerDied","Data":"fd842eae5943b90fcda246024796830eea809b913b01e6d1303818dc2e574e01"} Nov 02 21:30:28.760292 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.760129 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd842eae5943b90fcda246024796830eea809b913b01e6d1303818dc2e574e01" Nov 02 21:30:28.760292 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:28.760090 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4" Nov 02 21:30:51.685450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:51.685397 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:30:51.686209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:51.686167 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:30:52.703409 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:30:52.703362 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:30:54.492075 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.492035 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9"] Nov 02 21:30:54.492685 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.492663 5800 cpu_manager.go:414] "containerMap: removing container" podUID="740b0973-6474-4ae0-b9bf-c55033a946c9" containerName="revoke-keys" Nov 02 21:30:54.492786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.492687 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740b0973-6474-4ae0-b9bf-c55033a946c9" containerName="revoke-keys" Nov 02 21:30:54.492786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.492714 5800 cpu_manager.go:414] "containerMap: removing container" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:54.492786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.492722 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:54.492949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.492826 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="740b0973-6474-4ae0-b9bf-c55033a946c9" containerName="revoke-keys" Nov 02 21:30:54.492949 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.492846 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="918a1961-8e71-4eee-96ed-be60fd9dd8ff" containerName="cleanup" Nov 02 21:30:54.496200 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.496175 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.548946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.548909 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9"] Nov 02 21:30:54.558762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.558732 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dece7583-0afb-46b4-9371-21194080e8af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.558942 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.558781 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dece7583-0afb-46b4-9371-21194080e8af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.558942 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.558801 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgt8\" (UniqueName: \"kubernetes.io/projected/dece7583-0afb-46b4-9371-21194080e8af-kube-api-access-qjgt8\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.591554 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.591520 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:30:54.591554 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.591531 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:30:54.591554 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.591531 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:30:54.591797 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.591576 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:30:54.659570 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.659533 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgt8\" (UniqueName: \"kubernetes.io/projected/dece7583-0afb-46b4-9371-21194080e8af-kube-api-access-qjgt8\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.659880 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.659857 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dece7583-0afb-46b4-9371-21194080e8af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.659952 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.659934 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dece7583-0afb-46b4-9371-21194080e8af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.660599 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.660581 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dece7583-0afb-46b4-9371-21194080e8af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.662189 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.662172 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dece7583-0afb-46b4-9371-21194080e8af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.713038 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.712999 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgt8\" (UniqueName: \"kubernetes.io/projected/dece7583-0afb-46b4-9371-21194080e8af-kube-api-access-qjgt8\") pod \"maas-api-revoke-sub-c1e271a9-mdmp9\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:54.811916 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:54.811883 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:55.176114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:55.176088 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9"] Nov 02 21:30:55.177606 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:30:55.177576 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddece7583_0afb_46b4_9371_21194080e8af.slice/crio-67b4df6b75260de58644ee7803a0c290121016de1bc6993358096d4437e51e9e WatchSource:0}: Error finding container 67b4df6b75260de58644ee7803a0c290121016de1bc6993358096d4437e51e9e: Status 404 returned error can't find the container with id 67b4df6b75260de58644ee7803a0c290121016de1bc6993358096d4437e51e9e Nov 02 21:30:55.934430 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:55.934390 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" event={"ID":"dece7583-0afb-46b4-9371-21194080e8af","Type":"ContainerStarted","Data":"f69f56eeb51511f8a324c8248d331f56b3197510ce9da0ed3f52b56105649a4b"} Nov 02 21:30:55.934931 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:55.934455 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" event={"ID":"dece7583-0afb-46b4-9371-21194080e8af","Type":"ContainerStarted","Data":"67b4df6b75260de58644ee7803a0c290121016de1bc6993358096d4437e51e9e"} Nov 02 21:30:56.040447 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:56.040374 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" podStartSLOduration=2.040353384 podStartE2EDuration="2.040353384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:56.039755973 +0000 UTC m=+6154.556528558" watchObservedRunningTime="2026-11-02 21:30:56.040353384 +0000 UTC m=+6154.557125970" Nov 02 21:30:56.941770 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:56.941734 5800 generic.go:356] "Generic (PLEG): container finished" podID="dece7583-0afb-46b4-9371-21194080e8af" containerID="f69f56eeb51511f8a324c8248d331f56b3197510ce9da0ed3f52b56105649a4b" exitCode=0 Nov 02 21:30:56.942183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:56.941819 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" event={"ID":"dece7583-0afb-46b4-9371-21194080e8af","Type":"ContainerDied","Data":"f69f56eeb51511f8a324c8248d331f56b3197510ce9da0ed3f52b56105649a4b"} Nov 02 21:30:58.092843 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.092817 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:58.197251 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.197213 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dece7583-0afb-46b4-9371-21194080e8af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dece7583-0afb-46b4-9371-21194080e8af-maas-api-openshift-ca\") pod \"dece7583-0afb-46b4-9371-21194080e8af\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " Nov 02 21:30:58.197497 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.197335 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dece7583-0afb-46b4-9371-21194080e8af-kube-api-access-qjgt8\" (UniqueName: \"kubernetes.io/projected/dece7583-0afb-46b4-9371-21194080e8af-kube-api-access-qjgt8\") pod \"dece7583-0afb-46b4-9371-21194080e8af\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " Nov 02 21:30:58.197715 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.197515 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dece7583-0afb-46b4-9371-21194080e8af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dece7583-0afb-46b4-9371-21194080e8af-maas-api-xks-ca\") pod \"dece7583-0afb-46b4-9371-21194080e8af\" (UID: \"dece7583-0afb-46b4-9371-21194080e8af\") " Nov 02 21:30:58.197715 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.197668 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dece7583-0afb-46b4-9371-21194080e8af-maas-api-openshift-ca" pod "dece7583-0afb-46b4-9371-21194080e8af" (UID: "dece7583-0afb-46b4-9371-21194080e8af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:58.197950 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.197883 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dece7583-0afb-46b4-9371-21194080e8af-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:58.200107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.200077 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dece7583-0afb-46b4-9371-21194080e8af-kube-api-access-qjgt8" pod "dece7583-0afb-46b4-9371-21194080e8af" (UID: "dece7583-0afb-46b4-9371-21194080e8af"). InnerVolumeSpecName "kube-api-access-qjgt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:58.200371 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.200089 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dece7583-0afb-46b4-9371-21194080e8af-maas-api-xks-ca" pod "dece7583-0afb-46b4-9371-21194080e8af" (UID: "dece7583-0afb-46b4-9371-21194080e8af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:30:58.298810 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.298774 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dece7583-0afb-46b4-9371-21194080e8af-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:58.298810 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.298805 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgt8\" (UniqueName: \"kubernetes.io/projected/dece7583-0afb-46b4-9371-21194080e8af-kube-api-access-qjgt8\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:58.955120 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.955085 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" Nov 02 21:30:58.955350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.955123 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9" event={"ID":"dece7583-0afb-46b4-9371-21194080e8af","Type":"ContainerDied","Data":"67b4df6b75260de58644ee7803a0c290121016de1bc6993358096d4437e51e9e"} Nov 02 21:30:58.955350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:30:58.955207 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67b4df6b75260de58644ee7803a0c290121016de1bc6993358096d4437e51e9e" Nov 02 21:31:19.054967 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.054928 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8"] Nov 02 21:31:19.055540 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.055421 5800 cpu_manager.go:414] "containerMap: removing container" podUID="dece7583-0afb-46b4-9371-21194080e8af" containerName="revoke-keys" Nov 02 21:31:19.055540 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.055435 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dece7583-0afb-46b4-9371-21194080e8af" containerName="revoke-keys" Nov 02 21:31:19.055540 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.055516 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="dece7583-0afb-46b4-9371-21194080e8af" containerName="revoke-keys" Nov 02 21:31:19.058813 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.058791 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.108236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.108199 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:31:19.154557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.154523 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:31:19.154557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.154531 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:31:19.154896 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.154871 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:31:19.156997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.156974 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8"] Nov 02 21:31:19.248557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.248517 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhgk6\" (UniqueName: \"kubernetes.io/projected/cb1bfec8-6940-4da4-8602-2cb690c52b69-kube-api-access-zhgk6\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.248758 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.248564 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.248758 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.248686 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.349716 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.349624 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.349716 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.349700 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhgk6\" (UniqueName: \"kubernetes.io/projected/cb1bfec8-6940-4da4-8602-2cb690c52b69-kube-api-access-zhgk6\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.349987 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.349729 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.350419 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.350395 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.352316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.352268 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.402958 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.402922 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhgk6\" (UniqueName: \"kubernetes.io/projected/cb1bfec8-6940-4da4-8602-2cb690c52b69-kube-api-access-zhgk6\") pod \"maas-api-revoke-sub-2e1f5b5f-t2bt8\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:19.672864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:19.672767 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:20.039519 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:20.039492 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8"] Nov 02 21:31:20.041884 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:31:20.041856 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1bfec8_6940_4da4_8602_2cb690c52b69.slice/crio-4f31135c18dab16fe5bacf224603a574a0a6543af0f09ab10b6c30184e148dc7 WatchSource:0}: Error finding container 4f31135c18dab16fe5bacf224603a574a0a6543af0f09ab10b6c30184e148dc7: Status 404 returned error can't find the container with id 4f31135c18dab16fe5bacf224603a574a0a6543af0f09ab10b6c30184e148dc7 Nov 02 21:31:20.108118 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:20.108076 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" event={"ID":"cb1bfec8-6940-4da4-8602-2cb690c52b69","Type":"ContainerStarted","Data":"4f31135c18dab16fe5bacf224603a574a0a6543af0f09ab10b6c30184e148dc7"} Nov 02 21:31:21.114842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:21.114803 5800 generic.go:356] "Generic (PLEG): container finished" podID="cb1bfec8-6940-4da4-8602-2cb690c52b69" containerID="d58b3702786b34eafb9f896fc4b9b0ec4b0fbf1711ecf2042be908b755b39980" exitCode=0 Nov 02 21:31:21.115245 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:21.114884 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" event={"ID":"cb1bfec8-6940-4da4-8602-2cb690c52b69","Type":"ContainerDied","Data":"d58b3702786b34eafb9f896fc4b9b0ec4b0fbf1711ecf2042be908b755b39980"} Nov 02 21:31:22.265092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.265059 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:22.380447 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.380362 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1bfec8-6940-4da4-8602-2cb690c52b69-kube-api-access-zhgk6\" (UniqueName: \"kubernetes.io/projected/cb1bfec8-6940-4da4-8602-2cb690c52b69-kube-api-access-zhgk6\") pod \"cb1bfec8-6940-4da4-8602-2cb690c52b69\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " Nov 02 21:31:22.380447 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.380442 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-xks-ca\") pod \"cb1bfec8-6940-4da4-8602-2cb690c52b69\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " Nov 02 21:31:22.380701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.380576 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-openshift-ca\") pod \"cb1bfec8-6940-4da4-8602-2cb690c52b69\" (UID: \"cb1bfec8-6940-4da4-8602-2cb690c52b69\") " Nov 02 21:31:22.380986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.380957 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-openshift-ca" pod "cb1bfec8-6940-4da4-8602-2cb690c52b69" (UID: "cb1bfec8-6940-4da4-8602-2cb690c52b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:31:22.382643 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.382609 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1bfec8-6940-4da4-8602-2cb690c52b69-kube-api-access-zhgk6" pod "cb1bfec8-6940-4da4-8602-2cb690c52b69" (UID: "cb1bfec8-6940-4da4-8602-2cb690c52b69"). InnerVolumeSpecName "kube-api-access-zhgk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:31:22.382790 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.382662 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-xks-ca" pod "cb1bfec8-6940-4da4-8602-2cb690c52b69" (UID: "cb1bfec8-6940-4da4-8602-2cb690c52b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:31:22.481704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.481665 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:31:22.481704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.481700 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhgk6\" (UniqueName: \"kubernetes.io/projected/cb1bfec8-6940-4da4-8602-2cb690c52b69-kube-api-access-zhgk6\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:31:22.481704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:22.481711 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1bfec8-6940-4da4-8602-2cb690c52b69-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:31:22.696306 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:31:22.696200 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:31:23.127186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:23.127150 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" Nov 02 21:31:23.127406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:23.127188 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8" event={"ID":"cb1bfec8-6940-4da4-8602-2cb690c52b69","Type":"ContainerDied","Data":"4f31135c18dab16fe5bacf224603a574a0a6543af0f09ab10b6c30184e148dc7"} Nov 02 21:31:23.127406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:23.127242 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f31135c18dab16fe5bacf224603a574a0a6543af0f09ab10b6c30184e148dc7" Nov 02 21:31:27.093321 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:27.090409 5800 scope.go:122] "RemoveContainer" containerID="6233a88d12a754a41dcef1a2c5b7e03eb6392589d60b4b79ff8ca1df575b2a00" Nov 02 21:31:31.563916 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.563875 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9"] Nov 02 21:31:31.564494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.564353 5800 cpu_manager.go:414] "containerMap: removing container" podUID="cb1bfec8-6940-4da4-8602-2cb690c52b69" containerName="revoke-keys" Nov 02 21:31:31.564494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.564366 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1bfec8-6940-4da4-8602-2cb690c52b69" containerName="revoke-keys" Nov 02 21:31:31.564494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.564437 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1bfec8-6940-4da4-8602-2cb690c52b69" containerName="revoke-keys" Nov 02 21:31:31.569022 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.569001 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.633950 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.633914 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9"] Nov 02 21:31:31.662853 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.662813 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:31:31.662853 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.662823 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:31:31.663094 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.662885 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:31:31.664867 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.664850 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:31:31.668432 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.668405 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.668535 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.668445 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.668628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.668608 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46lj5\" (UniqueName: \"kubernetes.io/projected/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-kube-api-access-46lj5\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.770111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.770073 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lj5\" (UniqueName: \"kubernetes.io/projected/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-kube-api-access-46lj5\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.770315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.770167 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.770315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.770194 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.770930 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.770891 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.772622 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.772601 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.823674 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.823603 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46lj5\" (UniqueName: \"kubernetes.io/projected/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-kube-api-access-46lj5\") pod \"maas-api-revoke-sub-ec843018-9jhw9\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:31.882912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:31.882867 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:32.259352 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:32.256998 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9"] Nov 02 21:31:32.259352 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:31:32.258954 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c21b1f_0e6e_4726_8f08_41d1a9bc7e3a.slice/crio-9f8a9c1e3c8c2e95a7e1008392b343421d9bd568d2fbbdefb4e240c4c0496d87 WatchSource:0}: Error finding container 9f8a9c1e3c8c2e95a7e1008392b343421d9bd568d2fbbdefb4e240c4c0496d87: Status 404 returned error can't find the container with id 9f8a9c1e3c8c2e95a7e1008392b343421d9bd568d2fbbdefb4e240c4c0496d87 Nov 02 21:31:33.179489 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:31:33.179449 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c21b1f_0e6e_4726_8f08_41d1a9bc7e3a.slice/crio-conmon-5e91784e08674c7c26c60a96359611944d54621bde72654a65822ffdc80855e3.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:31:33.195773 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:33.195735 5800 generic.go:356] "Generic (PLEG): container finished" podID="35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" containerID="5e91784e08674c7c26c60a96359611944d54621bde72654a65822ffdc80855e3" exitCode=0 Nov 02 21:31:33.195984 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:33.195819 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" event={"ID":"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a","Type":"ContainerDied","Data":"5e91784e08674c7c26c60a96359611944d54621bde72654a65822ffdc80855e3"} Nov 02 21:31:33.195984 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:33.195915 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" event={"ID":"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a","Type":"ContainerStarted","Data":"9f8a9c1e3c8c2e95a7e1008392b343421d9bd568d2fbbdefb4e240c4c0496d87"} Nov 02 21:31:34.346428 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.346401 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:34.405493 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.405461 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-kube-api-access-46lj5\" (UniqueName: \"kubernetes.io/projected/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-kube-api-access-46lj5\") pod \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " Nov 02 21:31:34.405679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.405509 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-xks-ca\") pod \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " Nov 02 21:31:34.405679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.405562 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-openshift-ca\") pod \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\" (UID: \"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a\") " Nov 02 21:31:34.406035 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.406006 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-openshift-ca" pod "35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" (UID: "35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:31:34.407699 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.407670 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-xks-ca" pod "35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" (UID: "35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:31:34.407817 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.407732 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-kube-api-access-46lj5" pod "35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" (UID: "35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a"). InnerVolumeSpecName "kube-api-access-46lj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:31:34.507040 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.506999 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46lj5\" (UniqueName: \"kubernetes.io/projected/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-kube-api-access-46lj5\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:31:34.507229 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.507046 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:31:34.507229 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:34.507062 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:31:35.207565 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:35.207531 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" event={"ID":"35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a","Type":"ContainerDied","Data":"9f8a9c1e3c8c2e95a7e1008392b343421d9bd568d2fbbdefb4e240c4c0496d87"} Nov 02 21:31:35.207565 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:35.207552 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9" Nov 02 21:31:35.207808 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:31:35.207579 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8a9c1e3c8c2e95a7e1008392b343421d9bd568d2fbbdefb4e240c4c0496d87" Nov 02 21:31:52.720225 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:31:52.720183 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:32:05.981310 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:05.981203 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr"] Nov 02 21:32:05.981900 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:05.981873 5800 cpu_manager.go:414] "containerMap: removing container" podUID="35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" containerName="revoke-keys" Nov 02 21:32:05.981900 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:05.981896 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" containerName="revoke-keys" Nov 02 21:32:05.982052 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:05.982029 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" containerName="revoke-keys" Nov 02 21:32:05.985438 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:05.985415 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.035182 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.034745 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:32:06.038311 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.038258 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr"] Nov 02 21:32:06.055603 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.055570 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.055763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.055634 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gv4\" (UniqueName: \"kubernetes.io/projected/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-kube-api-access-c5gv4\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.055763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.055727 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.081059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.081025 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:32:06.081059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.081025 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:32:06.081059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.081025 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:32:06.156359 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.156323 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.156576 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.156486 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gv4\" (UniqueName: \"kubernetes.io/projected/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-kube-api-access-c5gv4\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.156576 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.156512 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.157180 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.157159 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.158824 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.158800 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.209968 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.209933 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gv4\" (UniqueName: \"kubernetes.io/projected/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-kube-api-access-c5gv4\") pod \"maas-api-revoke-sub-046e1343-r6hdr\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.298643 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.298609 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:06.664698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:06.664668 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr"] Nov 02 21:32:06.666460 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:32:06.666433 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f99808_7f19_4bb8_a8f0_eb87053fc4ad.slice/crio-89e2c30d488be4d46bc2ba7569e5f2d92006d21401264a2aa25cf15dd30bab81 WatchSource:0}: Error finding container 89e2c30d488be4d46bc2ba7569e5f2d92006d21401264a2aa25cf15dd30bab81: Status 404 returned error can't find the container with id 89e2c30d488be4d46bc2ba7569e5f2d92006d21401264a2aa25cf15dd30bab81 Nov 02 21:32:07.401630 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:07.401585 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" event={"ID":"92f99808-7f19-4bb8-a8f0-eb87053fc4ad","Type":"ContainerStarted","Data":"3f0ebdf1ea6406daad24588902baec03186b67a277303c361f11d9be81f8a116"} Nov 02 21:32:07.402160 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:07.401654 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" event={"ID":"92f99808-7f19-4bb8-a8f0-eb87053fc4ad","Type":"ContainerStarted","Data":"89e2c30d488be4d46bc2ba7569e5f2d92006d21401264a2aa25cf15dd30bab81"} Nov 02 21:32:07.508862 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:07.508806 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" podStartSLOduration=2.508790105 podStartE2EDuration="2.508790105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:07.506428038 +0000 UTC m=+6226.023200623" watchObservedRunningTime="2026-11-02 21:32:07.508790105 +0000 UTC m=+6226.025562690" Nov 02 21:32:08.409412 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:08.409372 5800 generic.go:356] "Generic (PLEG): container finished" podID="92f99808-7f19-4bb8-a8f0-eb87053fc4ad" containerID="3f0ebdf1ea6406daad24588902baec03186b67a277303c361f11d9be81f8a116" exitCode=0 Nov 02 21:32:08.409885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:08.409433 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" event={"ID":"92f99808-7f19-4bb8-a8f0-eb87053fc4ad","Type":"ContainerDied","Data":"3f0ebdf1ea6406daad24588902baec03186b67a277303c361f11d9be81f8a116"} Nov 02 21:32:09.561956 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.561927 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:09.694632 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.694540 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-xks-ca\") pod \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " Nov 02 21:32:09.694632 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.694582 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-kube-api-access-c5gv4\" (UniqueName: \"kubernetes.io/projected/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-kube-api-access-c5gv4\") pod \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " Nov 02 21:32:09.694895 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.694670 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-openshift-ca\") pod \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\" (UID: \"92f99808-7f19-4bb8-a8f0-eb87053fc4ad\") " Nov 02 21:32:09.695093 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.695063 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-openshift-ca" pod "92f99808-7f19-4bb8-a8f0-eb87053fc4ad" (UID: "92f99808-7f19-4bb8-a8f0-eb87053fc4ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:32:09.696860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.696825 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-xks-ca" pod "92f99808-7f19-4bb8-a8f0-eb87053fc4ad" (UID: "92f99808-7f19-4bb8-a8f0-eb87053fc4ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:32:09.696991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.696942 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-kube-api-access-c5gv4" pod "92f99808-7f19-4bb8-a8f0-eb87053fc4ad" (UID: "92f99808-7f19-4bb8-a8f0-eb87053fc4ad"). InnerVolumeSpecName "kube-api-access-c5gv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:32:09.795850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.795806 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:09.795850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.795843 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5gv4\" (UniqueName: \"kubernetes.io/projected/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-kube-api-access-c5gv4\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:09.796070 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:09.795861 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99808-7f19-4bb8-a8f0-eb87053fc4ad-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:10.421789 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:10.421751 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" Nov 02 21:32:10.421989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:10.421751 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr" event={"ID":"92f99808-7f19-4bb8-a8f0-eb87053fc4ad","Type":"ContainerDied","Data":"89e2c30d488be4d46bc2ba7569e5f2d92006d21401264a2aa25cf15dd30bab81"} Nov 02 21:32:10.421989 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:10.421886 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89e2c30d488be4d46bc2ba7569e5f2d92006d21401264a2aa25cf15dd30bab81" Nov 02 21:32:16.693939 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.693895 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq"] Nov 02 21:32:16.694598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.694318 5800 cpu_manager.go:414] "containerMap: removing container" podUID="92f99808-7f19-4bb8-a8f0-eb87053fc4ad" containerName="revoke-keys" Nov 02 21:32:16.694598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.694330 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f99808-7f19-4bb8-a8f0-eb87053fc4ad" containerName="revoke-keys" Nov 02 21:32:16.694598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.694416 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f99808-7f19-4bb8-a8f0-eb87053fc4ad" containerName="revoke-keys" Nov 02 21:32:16.697607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.697586 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.793074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.793038 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:32:16.793265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.793149 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:32:16.793265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.793169 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:32:16.793265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.793207 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:32:16.795670 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.795650 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq"] Nov 02 21:32:16.872153 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.872113 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.872389 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.872249 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.872389 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.872369 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r26j\" (UniqueName: \"kubernetes.io/projected/400cb5d5-d854-41f7-9b13-52ed357fa189-kube-api-access-2r26j\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.973255 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.973162 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.973255 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.973244 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.973561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.973342 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r26j\" (UniqueName: \"kubernetes.io/projected/400cb5d5-d854-41f7-9b13-52ed357fa189-kube-api-access-2r26j\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.973944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.973917 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:16.975676 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:16.975656 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:17.026211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:17.026173 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r26j\" (UniqueName: \"kubernetes.io/projected/400cb5d5-d854-41f7-9b13-52ed357fa189-kube-api-access-2r26j\") pod \"maas-api-revoke-sub-d3fba056-crjkq\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:17.311517 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:17.311481 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:17.677095 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:17.677068 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq"] Nov 02 21:32:17.679469 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:32:17.679440 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400cb5d5_d854_41f7_9b13_52ed357fa189.slice/crio-4bae896274b886491b3374668429419aa7294bba25765195c65cb6e8de0aee41 WatchSource:0}: Error finding container 4bae896274b886491b3374668429419aa7294bba25765195c65cb6e8de0aee41: Status 404 returned error can't find the container with id 4bae896274b886491b3374668429419aa7294bba25765195c65cb6e8de0aee41 Nov 02 21:32:18.471709 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:18.471671 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" event={"ID":"400cb5d5-d854-41f7-9b13-52ed357fa189","Type":"ContainerStarted","Data":"92417f11885178e68ae46de35426910bb30596d7c8de2215356a14928dccdbac"} Nov 02 21:32:18.472233 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:18.471735 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" event={"ID":"400cb5d5-d854-41f7-9b13-52ed357fa189","Type":"ContainerStarted","Data":"4bae896274b886491b3374668429419aa7294bba25765195c65cb6e8de0aee41"} Nov 02 21:32:18.579530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:18.579476 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" podStartSLOduration=2.579460346 podStartE2EDuration="2.579460346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:18.576563596 +0000 UTC m=+6237.093336180" watchObservedRunningTime="2026-11-02 21:32:18.579460346 +0000 UTC m=+6237.096232931" Nov 02 21:32:19.481495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:19.481453 5800 generic.go:356] "Generic (PLEG): container finished" podID="400cb5d5-d854-41f7-9b13-52ed357fa189" containerID="92417f11885178e68ae46de35426910bb30596d7c8de2215356a14928dccdbac" exitCode=0 Nov 02 21:32:19.482056 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:19.481539 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" event={"ID":"400cb5d5-d854-41f7-9b13-52ed357fa189","Type":"ContainerDied","Data":"92417f11885178e68ae46de35426910bb30596d7c8de2215356a14928dccdbac"} Nov 02 21:32:21.067307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.066709 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:21.133308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.130613 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400cb5d5-d854-41f7-9b13-52ed357fa189-kube-api-access-2r26j\" (UniqueName: \"kubernetes.io/projected/400cb5d5-d854-41f7-9b13-52ed357fa189-kube-api-access-2r26j\") pod \"400cb5d5-d854-41f7-9b13-52ed357fa189\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " Nov 02 21:32:21.133308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.130683 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-openshift-ca\") pod \"400cb5d5-d854-41f7-9b13-52ed357fa189\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " Nov 02 21:32:21.133308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.130826 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-xks-ca\") pod \"400cb5d5-d854-41f7-9b13-52ed357fa189\" (UID: \"400cb5d5-d854-41f7-9b13-52ed357fa189\") " Nov 02 21:32:21.133308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.131689 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-openshift-ca" pod "400cb5d5-d854-41f7-9b13-52ed357fa189" (UID: "400cb5d5-d854-41f7-9b13-52ed357fa189"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:32:21.156531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.151823 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400cb5d5-d854-41f7-9b13-52ed357fa189-kube-api-access-2r26j" pod "400cb5d5-d854-41f7-9b13-52ed357fa189" (UID: "400cb5d5-d854-41f7-9b13-52ed357fa189"). InnerVolumeSpecName "kube-api-access-2r26j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:32:21.179306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.166479 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-xks-ca" pod "400cb5d5-d854-41f7-9b13-52ed357fa189" (UID: "400cb5d5-d854-41f7-9b13-52ed357fa189"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:32:21.253370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.239956 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r26j\" (UniqueName: \"kubernetes.io/projected/400cb5d5-d854-41f7-9b13-52ed357fa189-kube-api-access-2r26j\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:21.253370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.240004 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:21.253370 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.240021 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400cb5d5-d854-41f7-9b13-52ed357fa189-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:21.511762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.511665 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" event={"ID":"400cb5d5-d854-41f7-9b13-52ed357fa189","Type":"ContainerDied","Data":"4bae896274b886491b3374668429419aa7294bba25765195c65cb6e8de0aee41"} Nov 02 21:32:21.511762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.511722 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bae896274b886491b3374668429419aa7294bba25765195c65cb6e8de0aee41" Nov 02 21:32:21.511762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.511675 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq" Nov 02 21:32:21.690537 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.690494 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:32:21.690856 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:21.690808 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:32:22.697714 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:32:22.697677 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:32:24.042411 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:24.042375 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw"] Nov 02 21:32:24.088939 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:24.088901 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-b8gzw"] Nov 02 21:32:25.621083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:25.621045 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b81348-1e20-45b8-b7b8-e830b4345cec" path="/var/lib/kubelet/pods/63b81348-1e20-45b8-b7b8-e830b4345cec/volumes" Nov 02 21:32:44.042528 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:44.042493 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs"] Nov 02 21:32:44.088786 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:44.088755 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t4txs"] Nov 02 21:32:45.620564 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:45.620528 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d292af-6d43-4dd1-9525-5d478fb7e15e" path="/var/lib/kubelet/pods/60d292af-6d43-4dd1-9525-5d478fb7e15e/volumes" Nov 02 21:32:51.243871 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.243832 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68"] Nov 02 21:32:51.244299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.244256 5800 cpu_manager.go:414] "containerMap: removing container" podUID="400cb5d5-d854-41f7-9b13-52ed357fa189" containerName="revoke-keys" Nov 02 21:32:51.244299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.244268 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400cb5d5-d854-41f7-9b13-52ed357fa189" containerName="revoke-keys" Nov 02 21:32:51.244395 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.244389 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="400cb5d5-d854-41f7-9b13-52ed357fa189" containerName="revoke-keys" Nov 02 21:32:51.247762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.247745 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.300006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.299971 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68"] Nov 02 21:32:51.306579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.306543 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4rf\" (UniqueName: \"kubernetes.io/projected/ea56e9d5-87a4-4d75-987a-0074a573bebf-kube-api-access-6v4rf\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.306769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.306587 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.306769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.306613 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.343185 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.343131 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:32:51.343185 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.343172 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:32:51.343487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.343312 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:32:51.343487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.343351 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:32:51.407542 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.407507 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.407725 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.407548 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.407799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.407721 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4rf\" (UniqueName: \"kubernetes.io/projected/ea56e9d5-87a4-4d75-987a-0074a573bebf-kube-api-access-6v4rf\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.408385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.408356 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.409950 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.409932 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.460376 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.460342 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4rf\" (UniqueName: \"kubernetes.io/projected/ea56e9d5-87a4-4d75-987a-0074a573bebf-kube-api-access-6v4rf\") pod \"maas-api-revoke-sub-ae594979-2wj68\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.560940 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.560901 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:51.926792 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:51.926766 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68"] Nov 02 21:32:51.927996 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:32:51.927960 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea56e9d5_87a4_4d75_987a_0074a573bebf.slice/crio-de5b134982986e178cb46233e165f8490dcb0a5778b479e35f411f334a5ff56f WatchSource:0}: Error finding container de5b134982986e178cb46233e165f8490dcb0a5778b479e35f411f334a5ff56f: Status 404 returned error can't find the container with id de5b134982986e178cb46233e165f8490dcb0a5778b479e35f411f334a5ff56f Nov 02 21:32:52.697169 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:32:52.697133 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:32:52.703073 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:52.703043 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" event={"ID":"ea56e9d5-87a4-4d75-987a-0074a573bebf","Type":"ContainerStarted","Data":"a23d1b77b7a57a2190a674b721d584fa30c4b696187334ab47ddbe1cad03390f"} Nov 02 21:32:52.703249 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:52.703091 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" event={"ID":"ea56e9d5-87a4-4d75-987a-0074a573bebf","Type":"ContainerStarted","Data":"de5b134982986e178cb46233e165f8490dcb0a5778b479e35f411f334a5ff56f"} Nov 02 21:32:52.846976 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:52.846920 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" podStartSLOduration=1.846907024 podStartE2EDuration="1.846907024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:52.808393874 +0000 UTC m=+6271.325166460" watchObservedRunningTime="2026-11-02 21:32:52.846907024 +0000 UTC m=+6271.363679605" Nov 02 21:32:53.710399 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:53.710363 5800 generic.go:356] "Generic (PLEG): container finished" podID="ea56e9d5-87a4-4d75-987a-0074a573bebf" containerID="a23d1b77b7a57a2190a674b721d584fa30c4b696187334ab47ddbe1cad03390f" exitCode=0 Nov 02 21:32:53.710818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:53.710454 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" event={"ID":"ea56e9d5-87a4-4d75-987a-0074a573bebf","Type":"ContainerDied","Data":"a23d1b77b7a57a2190a674b721d584fa30c4b696187334ab47ddbe1cad03390f"} Nov 02 21:32:54.859817 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:54.859792 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:54.946183 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:54.946142 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea56e9d5-87a4-4d75-987a-0074a573bebf-kube-api-access-6v4rf\" (UniqueName: \"kubernetes.io/projected/ea56e9d5-87a4-4d75-987a-0074a573bebf-kube-api-access-6v4rf\") pod \"ea56e9d5-87a4-4d75-987a-0074a573bebf\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " Nov 02 21:32:54.946406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:54.946314 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-openshift-ca\") pod \"ea56e9d5-87a4-4d75-987a-0074a573bebf\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " Nov 02 21:32:54.946406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:54.946343 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-xks-ca\") pod \"ea56e9d5-87a4-4d75-987a-0074a573bebf\" (UID: \"ea56e9d5-87a4-4d75-987a-0074a573bebf\") " Nov 02 21:32:54.946710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:54.946683 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-openshift-ca" pod "ea56e9d5-87a4-4d75-987a-0074a573bebf" (UID: "ea56e9d5-87a4-4d75-987a-0074a573bebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:32:54.948430 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:54.948401 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-xks-ca" pod "ea56e9d5-87a4-4d75-987a-0074a573bebf" (UID: "ea56e9d5-87a4-4d75-987a-0074a573bebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:32:54.948430 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:54.948408 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea56e9d5-87a4-4d75-987a-0074a573bebf-kube-api-access-6v4rf" pod "ea56e9d5-87a4-4d75-987a-0074a573bebf" (UID: "ea56e9d5-87a4-4d75-987a-0074a573bebf"). InnerVolumeSpecName "kube-api-access-6v4rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:32:55.047291 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:55.047232 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4rf\" (UniqueName: \"kubernetes.io/projected/ea56e9d5-87a4-4d75-987a-0074a573bebf-kube-api-access-6v4rf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:55.047291 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:55.047269 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:55.047291 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:55.047297 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea56e9d5-87a4-4d75-987a-0074a573bebf-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:32:55.722766 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:55.722734 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" Nov 02 21:32:55.722766 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:55.722738 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68" event={"ID":"ea56e9d5-87a4-4d75-987a-0074a573bebf","Type":"ContainerDied","Data":"de5b134982986e178cb46233e165f8490dcb0a5778b479e35f411f334a5ff56f"} Nov 02 21:32:55.722991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:32:55.722794 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de5b134982986e178cb46233e165f8490dcb0a5778b479e35f411f334a5ff56f" Nov 02 21:33:06.042896 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:06.042857 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8"] Nov 02 21:33:06.089527 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:06.089482 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-941f5cfd-5prz8"] Nov 02 21:33:07.620731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:07.620695 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1424fad-385b-4927-8378-0652243a53d3" path="/var/lib/kubelet/pods/e1424fad-385b-4927-8378-0652243a53d3/volumes" Nov 02 21:33:09.762306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.760711 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9"] Nov 02 21:33:09.762306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.761538 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ea56e9d5-87a4-4d75-987a-0074a573bebf" containerName="revoke-keys" Nov 02 21:33:09.762306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.761560 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea56e9d5-87a4-4d75-987a-0074a573bebf" containerName="revoke-keys" Nov 02 21:33:09.762306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.761725 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea56e9d5-87a4-4d75-987a-0074a573bebf" containerName="revoke-keys" Nov 02 21:33:09.767309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.766078 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:09.817450 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.817414 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9"] Nov 02 21:33:09.877306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.861679 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:33:09.877614 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.869463 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:09.877746 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.869622 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:09.877842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.869704 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:09.901104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.901064 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvr7\" (UniqueName: \"kubernetes.io/projected/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-kube-api-access-ljvr7\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:09.901583 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.901560 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:09.901769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:09.901751 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.002945 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.002904 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvr7\" (UniqueName: \"kubernetes.io/projected/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-kube-api-access-ljvr7\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.003577 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.003554 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.004182 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.004160 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.005232 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.005057 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.006467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.006436 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.088255 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.088171 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvr7\" (UniqueName: \"kubernetes.io/projected/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-kube-api-access-ljvr7\") pod \"maas-api-revoke-sub-a4aef7d7-6qnc9\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.106837 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.106799 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:10.705785 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.705585 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9"] Nov 02 21:33:10.815654 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:10.815616 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" event={"ID":"d961c60e-859d-42f3-8cf6-c1dff0cd5e07","Type":"ContainerStarted","Data":"2a6ef2198199930ffe5da4e4ac724b2107e97580569a8716b54c6231f61d37e2"} Nov 02 21:33:11.824026 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:11.823992 5800 generic.go:356] "Generic (PLEG): container finished" podID="d961c60e-859d-42f3-8cf6-c1dff0cd5e07" containerID="1e16a957d72815f031d9d49e7ffd95e306e274e10393979761be37780b4493b8" exitCode=0 Nov 02 21:33:11.824443 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:11.824073 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" event={"ID":"d961c60e-859d-42f3-8cf6-c1dff0cd5e07","Type":"ContainerDied","Data":"1e16a957d72815f031d9d49e7ffd95e306e274e10393979761be37780b4493b8"} Nov 02 21:33:12.979115 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:12.979077 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:13.085759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.085660 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-kube-api-access-ljvr7\" (UniqueName: \"kubernetes.io/projected/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-kube-api-access-ljvr7\") pod \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " Nov 02 21:33:13.085759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.085745 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-xks-ca\") pod \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " Nov 02 21:33:13.086007 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.085863 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-openshift-ca\") pod \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\" (UID: \"d961c60e-859d-42f3-8cf6-c1dff0cd5e07\") " Nov 02 21:33:13.086302 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.086254 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-openshift-ca" pod "d961c60e-859d-42f3-8cf6-c1dff0cd5e07" (UID: "d961c60e-859d-42f3-8cf6-c1dff0cd5e07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:13.087896 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.087870 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-xks-ca" pod "d961c60e-859d-42f3-8cf6-c1dff0cd5e07" (UID: "d961c60e-859d-42f3-8cf6-c1dff0cd5e07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:13.088007 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.087916 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-kube-api-access-ljvr7" pod "d961c60e-859d-42f3-8cf6-c1dff0cd5e07" (UID: "d961c60e-859d-42f3-8cf6-c1dff0cd5e07"). InnerVolumeSpecName "kube-api-access-ljvr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:13.187521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.187482 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvr7\" (UniqueName: \"kubernetes.io/projected/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-kube-api-access-ljvr7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:13.187521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.187518 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:13.187521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.187528 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d961c60e-859d-42f3-8cf6-c1dff0cd5e07-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:13.839246 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.839216 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" Nov 02 21:33:13.839246 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.839236 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9" event={"ID":"d961c60e-859d-42f3-8cf6-c1dff0cd5e07","Type":"ContainerDied","Data":"2a6ef2198199930ffe5da4e4ac724b2107e97580569a8716b54c6231f61d37e2"} Nov 02 21:33:13.839492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:13.839312 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6ef2198199930ffe5da4e4ac724b2107e97580569a8716b54c6231f61d37e2" Nov 02 21:33:22.696631 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:33:22.696586 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:33:22.777759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:22.777729 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:33:22.789701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:22.789660 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:33:27.129468 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:27.129432 5800 scope.go:122] "RemoveContainer" containerID="d9c9bd5cc65f3141398b0799b7c619d2457707faa48dbccaacf98d9ae883017a" Nov 02 21:33:30.826872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.826619 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp"] Nov 02 21:33:30.828025 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.827992 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d961c60e-859d-42f3-8cf6-c1dff0cd5e07" containerName="revoke-keys" Nov 02 21:33:30.828158 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.828025 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d961c60e-859d-42f3-8cf6-c1dff0cd5e07" containerName="revoke-keys" Nov 02 21:33:30.828239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.828156 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d961c60e-859d-42f3-8cf6-c1dff0cd5e07" containerName="revoke-keys" Nov 02 21:33:30.833425 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.833398 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.870495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.870452 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.870900 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.870855 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.871082 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.870944 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98hv\" (UniqueName: \"kubernetes.io/projected/26398c00-2b3d-46f7-9db7-de7912ab0479-kube-api-access-m98hv\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.887199 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.887154 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp"] Nov 02 21:33:30.929378 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.929340 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:30.929634 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.929358 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:30.929823 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.929395 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:30.929940 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.929543 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:33:30.971444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.971400 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.971444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.971451 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.971739 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.971506 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98hv\" (UniqueName: \"kubernetes.io/projected/26398c00-2b3d-46f7-9db7-de7912ab0479-kube-api-access-m98hv\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.972385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.972357 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:30.974639 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:30.974603 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:31.024724 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:31.024691 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98hv\" (UniqueName: \"kubernetes.io/projected/26398c00-2b3d-46f7-9db7-de7912ab0479-kube-api-access-m98hv\") pod \"maas-api-revoke-sub-79805ca9-cxmpp\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:31.173021 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:31.172651 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:31.674213 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:33:31.674159 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26398c00_2b3d_46f7_9db7_de7912ab0479.slice/crio-316d79e0f3c28506299e0d1c8198edba02281cdcce11ec1239c1c4946c6c0f13 WatchSource:0}: Error finding container 316d79e0f3c28506299e0d1c8198edba02281cdcce11ec1239c1c4946c6c0f13: Status 404 returned error can't find the container with id 316d79e0f3c28506299e0d1c8198edba02281cdcce11ec1239c1c4946c6c0f13 Nov 02 21:33:31.712603 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:31.712323 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp"] Nov 02 21:33:31.988852 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:31.978416 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" event={"ID":"26398c00-2b3d-46f7-9db7-de7912ab0479","Type":"ContainerStarted","Data":"316d79e0f3c28506299e0d1c8198edba02281cdcce11ec1239c1c4946c6c0f13"} Nov 02 21:33:32.987338 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:32.987273 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" event={"ID":"26398c00-2b3d-46f7-9db7-de7912ab0479","Type":"ContainerStarted","Data":"4043db827e984c9e5faa0b9f4059ea7f4aa7cda30cc702a42874b5c668e69ddf"} Nov 02 21:33:33.093541 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:33.093481 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" podStartSLOduration=3.09346746 podStartE2EDuration="3.09346746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:33.091819699 +0000 UTC m=+6311.608592284" watchObservedRunningTime="2026-11-02 21:33:33.09346746 +0000 UTC m=+6311.610240044" Nov 02 21:33:33.995695 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:33.995652 5800 generic.go:356] "Generic (PLEG): container finished" podID="26398c00-2b3d-46f7-9db7-de7912ab0479" containerID="4043db827e984c9e5faa0b9f4059ea7f4aa7cda30cc702a42874b5c668e69ddf" exitCode=0 Nov 02 21:33:33.995914 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:33.995713 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" event={"ID":"26398c00-2b3d-46f7-9db7-de7912ab0479","Type":"ContainerDied","Data":"4043db827e984c9e5faa0b9f4059ea7f4aa7cda30cc702a42874b5c668e69ddf"} Nov 02 21:33:35.144146 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.144118 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:35.236464 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.236427 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-openshift-ca\") pod \"26398c00-2b3d-46f7-9db7-de7912ab0479\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " Nov 02 21:33:35.236672 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.236532 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26398c00-2b3d-46f7-9db7-de7912ab0479-kube-api-access-m98hv\" (UniqueName: \"kubernetes.io/projected/26398c00-2b3d-46f7-9db7-de7912ab0479-kube-api-access-m98hv\") pod \"26398c00-2b3d-46f7-9db7-de7912ab0479\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " Nov 02 21:33:35.236672 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.236596 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-xks-ca\") pod \"26398c00-2b3d-46f7-9db7-de7912ab0479\" (UID: \"26398c00-2b3d-46f7-9db7-de7912ab0479\") " Nov 02 21:33:35.236882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.236855 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-openshift-ca" pod "26398c00-2b3d-46f7-9db7-de7912ab0479" (UID: "26398c00-2b3d-46f7-9db7-de7912ab0479"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:35.238699 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.238667 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-xks-ca" pod "26398c00-2b3d-46f7-9db7-de7912ab0479" (UID: "26398c00-2b3d-46f7-9db7-de7912ab0479"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:35.238699 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.238680 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26398c00-2b3d-46f7-9db7-de7912ab0479-kube-api-access-m98hv" pod "26398c00-2b3d-46f7-9db7-de7912ab0479" (UID: "26398c00-2b3d-46f7-9db7-de7912ab0479"). InnerVolumeSpecName "kube-api-access-m98hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:35.338107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.338014 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:35.338107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.338049 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m98hv\" (UniqueName: \"kubernetes.io/projected/26398c00-2b3d-46f7-9db7-de7912ab0479-kube-api-access-m98hv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:35.338107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:35.338061 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26398c00-2b3d-46f7-9db7-de7912ab0479-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:36.018572 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:36.018538 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" Nov 02 21:33:36.018789 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:36.018560 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp" event={"ID":"26398c00-2b3d-46f7-9db7-de7912ab0479","Type":"ContainerDied","Data":"316d79e0f3c28506299e0d1c8198edba02281cdcce11ec1239c1c4946c6c0f13"} Nov 02 21:33:36.018789 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:36.018710 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="316d79e0f3c28506299e0d1c8198edba02281cdcce11ec1239c1c4946c6c0f13" Nov 02 21:33:48.130899 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.130856 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w"] Nov 02 21:33:48.131552 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.131525 5800 cpu_manager.go:414] "containerMap: removing container" podUID="26398c00-2b3d-46f7-9db7-de7912ab0479" containerName="revoke-keys" Nov 02 21:33:48.131676 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.131552 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26398c00-2b3d-46f7-9db7-de7912ab0479" containerName="revoke-keys" Nov 02 21:33:48.131754 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.131713 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="26398c00-2b3d-46f7-9db7-de7912ab0479" containerName="revoke-keys" Nov 02 21:33:48.135594 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.135565 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.192114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.192065 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w"] Nov 02 21:33:48.196036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.196000 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.196226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.196067 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmnq\" (UniqueName: \"kubernetes.io/projected/914a01d1-fefe-4d1c-aec9-7414581968fa-kube-api-access-9gmnq\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.196226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.196097 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.228587 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.228551 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:48.230612 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.230581 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:48.230612 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.230591 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:33:48.230612 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.230599 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:48.296996 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.296957 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.296996 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.297002 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmnq\" (UniqueName: \"kubernetes.io/projected/914a01d1-fefe-4d1c-aec9-7414581968fa-kube-api-access-9gmnq\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.297219 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.297020 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.297657 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.297635 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.299430 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.299411 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.350132 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.350101 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmnq\" (UniqueName: \"kubernetes.io/projected/914a01d1-fefe-4d1c-aec9-7414581968fa-kube-api-access-9gmnq\") pod \"maas-api-revoke-sub-0b7e8edb-hvv5w\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.457615 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.457532 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:48.824538 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:48.824505 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w"] Nov 02 21:33:48.826611 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:33:48.826573 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914a01d1_fefe_4d1c_aec9_7414581968fa.slice/crio-310054445b80280a6355a6a1ae6a364ece4656d8e78aeb3f3decb0a8900e1a33 WatchSource:0}: Error finding container 310054445b80280a6355a6a1ae6a364ece4656d8e78aeb3f3decb0a8900e1a33: Status 404 returned error can't find the container with id 310054445b80280a6355a6a1ae6a364ece4656d8e78aeb3f3decb0a8900e1a33 Nov 02 21:33:49.112483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:49.112402 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" event={"ID":"914a01d1-fefe-4d1c-aec9-7414581968fa","Type":"ContainerStarted","Data":"310054445b80280a6355a6a1ae6a364ece4656d8e78aeb3f3decb0a8900e1a33"} Nov 02 21:33:50.119679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:50.119636 5800 generic.go:356] "Generic (PLEG): container finished" podID="914a01d1-fefe-4d1c-aec9-7414581968fa" containerID="3fac5860f9581e8bd2fdf6ff4935eb197dcdec9f60421ae953dc8c3f9e84b6b6" exitCode=0 Nov 02 21:33:50.120079 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:50.119727 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" event={"ID":"914a01d1-fefe-4d1c-aec9-7414581968fa","Type":"ContainerDied","Data":"3fac5860f9581e8bd2fdf6ff4935eb197dcdec9f60421ae953dc8c3f9e84b6b6"} Nov 02 21:33:51.272922 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.272897 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:51.329175 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.329149 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914a01d1-fefe-4d1c-aec9-7414581968fa-kube-api-access-9gmnq\" (UniqueName: \"kubernetes.io/projected/914a01d1-fefe-4d1c-aec9-7414581968fa-kube-api-access-9gmnq\") pod \"914a01d1-fefe-4d1c-aec9-7414581968fa\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " Nov 02 21:33:51.329369 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.329208 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-openshift-ca\") pod \"914a01d1-fefe-4d1c-aec9-7414581968fa\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " Nov 02 21:33:51.329369 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.329240 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-xks-ca\") pod \"914a01d1-fefe-4d1c-aec9-7414581968fa\" (UID: \"914a01d1-fefe-4d1c-aec9-7414581968fa\") " Nov 02 21:33:51.329616 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.329588 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-openshift-ca" pod "914a01d1-fefe-4d1c-aec9-7414581968fa" (UID: "914a01d1-fefe-4d1c-aec9-7414581968fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:33:51.331434 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.331413 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914a01d1-fefe-4d1c-aec9-7414581968fa-kube-api-access-9gmnq" pod "914a01d1-fefe-4d1c-aec9-7414581968fa" (UID: "914a01d1-fefe-4d1c-aec9-7414581968fa"). InnerVolumeSpecName "kube-api-access-9gmnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:33:51.331510 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.331441 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-xks-ca" pod "914a01d1-fefe-4d1c-aec9-7414581968fa" (UID: "914a01d1-fefe-4d1c-aec9-7414581968fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:33:51.430947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.430865 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gmnq\" (UniqueName: \"kubernetes.io/projected/914a01d1-fefe-4d1c-aec9-7414581968fa-kube-api-access-9gmnq\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:51.430947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.430901 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:51.430947 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:51.430917 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914a01d1-fefe-4d1c-aec9-7414581968fa-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:33:52.133091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:52.133058 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" Nov 02 21:33:52.133091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:52.133067 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w" event={"ID":"914a01d1-fefe-4d1c-aec9-7414581968fa","Type":"ContainerDied","Data":"310054445b80280a6355a6a1ae6a364ece4656d8e78aeb3f3decb0a8900e1a33"} Nov 02 21:33:52.133395 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:52.133111 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310054445b80280a6355a6a1ae6a364ece4656d8e78aeb3f3decb0a8900e1a33" Nov 02 21:33:52.698520 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:33:52.698481 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:33:59.431271 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.431227 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt"] Nov 02 21:33:59.431933 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.431909 5800 cpu_manager.go:414] "containerMap: removing container" podUID="914a01d1-fefe-4d1c-aec9-7414581968fa" containerName="revoke-keys" Nov 02 21:33:59.432014 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.431932 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914a01d1-fefe-4d1c-aec9-7414581968fa" containerName="revoke-keys" Nov 02 21:33:59.432098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.432083 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="914a01d1-fefe-4d1c-aec9-7414581968fa" containerName="revoke-keys" Nov 02 21:33:59.435519 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.435498 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.486901 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.486866 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt"] Nov 02 21:33:59.516999 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.516960 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.517177 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.517095 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.517177 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.517140 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lll\" (UniqueName: \"kubernetes.io/projected/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-kube-api-access-w2lll\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.529738 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.529705 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:33:59.529898 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.529764 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:33:59.529898 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.529765 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:33:59.530475 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.530455 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:33:59.617803 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.617765 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.617993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.617826 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lll\" (UniqueName: \"kubernetes.io/projected/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-kube-api-access-w2lll\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.617993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.617955 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.618554 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.618536 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.620558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.620530 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.671526 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.671491 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lll\" (UniqueName: \"kubernetes.io/projected/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-kube-api-access-w2lll\") pod \"maas-api-revoke-sub-3d4156f1-j7znt\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:33:59.752808 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:33:59.752781 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:34:00.120140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:00.120114 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt"] Nov 02 21:34:00.122570 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:34:00.122538 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcd4cf4_ad4f_4292_aba1_8dd43282b2c8.slice/crio-d42fa748dd97f63326a8c1aa498264fccb14e5964799a72ef5a9f30703486eac WatchSource:0}: Error finding container d42fa748dd97f63326a8c1aa498264fccb14e5964799a72ef5a9f30703486eac: Status 404 returned error can't find the container with id d42fa748dd97f63326a8c1aa498264fccb14e5964799a72ef5a9f30703486eac Nov 02 21:34:00.181862 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:00.181829 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" event={"ID":"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8","Type":"ContainerStarted","Data":"d42fa748dd97f63326a8c1aa498264fccb14e5964799a72ef5a9f30703486eac"} Nov 02 21:34:01.189507 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:01.189418 5800 generic.go:356] "Generic (PLEG): container finished" podID="0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" containerID="f7c9e4bf389a34118b3166bf6c819695cf680d802ee3571c28aa36138962638c" exitCode=0 Nov 02 21:34:01.189507 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:01.189482 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" event={"ID":"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8","Type":"ContainerDied","Data":"f7c9e4bf389a34118b3166bf6c819695cf680d802ee3571c28aa36138962638c"} Nov 02 21:34:02.354226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.354200 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:34:02.450727 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.450642 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-openshift-ca\") pod \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " Nov 02 21:34:02.450894 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.450753 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-kube-api-access-w2lll\" (UniqueName: \"kubernetes.io/projected/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-kube-api-access-w2lll\") pod \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " Nov 02 21:34:02.450894 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.450798 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-xks-ca\") pod \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\" (UID: \"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8\") " Nov 02 21:34:02.451027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.450997 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-openshift-ca" pod "0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" (UID: "0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:02.451115 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.451098 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:02.453012 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.452980 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-xks-ca" pod "0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" (UID: "0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:02.453137 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.453050 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-kube-api-access-w2lll" pod "0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" (UID: "0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8"). InnerVolumeSpecName "kube-api-access-w2lll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:02.551764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.551734 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2lll\" (UniqueName: \"kubernetes.io/projected/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-kube-api-access-w2lll\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:02.551764 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:02.551761 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:03.203614 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:03.203581 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" Nov 02 21:34:03.203614 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:03.203595 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt" event={"ID":"0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8","Type":"ContainerDied","Data":"d42fa748dd97f63326a8c1aa498264fccb14e5964799a72ef5a9f30703486eac"} Nov 02 21:34:03.203860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:03.203665 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d42fa748dd97f63326a8c1aa498264fccb14e5964799a72ef5a9f30703486eac" Nov 02 21:34:22.697089 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:34:22.697050 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:34:22.984301 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:22.984199 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt"] Nov 02 21:34:23.030524 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.030474 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-j7znt"] Nov 02 21:34:23.099335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.099305 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr"] Nov 02 21:34:23.099722 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.099710 5800 cpu_manager.go:414] "containerMap: removing container" podUID="0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" containerName="revoke-keys" Nov 02 21:34:23.099778 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.099724 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" containerName="revoke-keys" Nov 02 21:34:23.099842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.099831 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" containerName="revoke-keys" Nov 02 21:34:23.102436 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.102415 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.103308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.103270 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.103425 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.103404 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.103485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.103447 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdbf\" (UniqueName: \"kubernetes.io/projected/7d036e07-e65e-4b88-81ea-57c1f12e97d4-kube-api-access-9gdbf\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.151680 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.151646 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:34:23.151992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.151964 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:34:23.152385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.152131 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:34:23.152385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.152305 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:34:23.155074 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.155050 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr"] Nov 02 21:34:23.204377 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.204344 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.204565 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.204417 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.204565 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.204465 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdbf\" (UniqueName: \"kubernetes.io/projected/7d036e07-e65e-4b88-81ea-57c1f12e97d4-kube-api-access-9gdbf\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.205113 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.205087 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.206792 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.206761 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.257720 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.257693 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdbf\" (UniqueName: \"kubernetes.io/projected/7d036e07-e65e-4b88-81ea-57c1f12e97d4-kube-api-access-9gdbf\") pod \"maas-api-revoke-sub-3d4156f1-6wnkr\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.415347 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.415297 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:23.621874 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.621791 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8" path="/var/lib/kubelet/pods/0fcd4cf4-ad4f-4292-aba1-8dd43282b2c8/volumes" Nov 02 21:34:23.793045 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:23.791092 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr"] Nov 02 21:34:23.793045 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:34:23.792442 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d036e07_e65e_4b88_81ea_57c1f12e97d4.slice/crio-bf9077845f6d05002d70bbd565466c1dc9b888ff98a777b6f28d089df67ed9e7 WatchSource:0}: Error finding container bf9077845f6d05002d70bbd565466c1dc9b888ff98a777b6f28d089df67ed9e7: Status 404 returned error can't find the container with id bf9077845f6d05002d70bbd565466c1dc9b888ff98a777b6f28d089df67ed9e7 Nov 02 21:34:24.387463 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:24.387370 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" event={"ID":"7d036e07-e65e-4b88-81ea-57c1f12e97d4","Type":"ContainerStarted","Data":"f2681184fb060b1c886c8733d5cf10d9c28ccad75a5a47ab6128b5cfae594567"} Nov 02 21:34:24.387463 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:24.387421 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" event={"ID":"7d036e07-e65e-4b88-81ea-57c1f12e97d4","Type":"ContainerStarted","Data":"bf9077845f6d05002d70bbd565466c1dc9b888ff98a777b6f28d089df67ed9e7"} Nov 02 21:34:24.493582 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:24.493515 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" podStartSLOduration=1.4935002929999999 podStartE2EDuration="1.493500293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:24.492581133 +0000 UTC m=+6363.009353719" watchObservedRunningTime="2026-11-02 21:34:24.493500293 +0000 UTC m=+6363.010272874" Nov 02 21:34:25.394269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:25.394232 5800 generic.go:356] "Generic (PLEG): container finished" podID="7d036e07-e65e-4b88-81ea-57c1f12e97d4" containerID="f2681184fb060b1c886c8733d5cf10d9c28ccad75a5a47ab6128b5cfae594567" exitCode=0 Nov 02 21:34:25.394676 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:25.394324 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" event={"ID":"7d036e07-e65e-4b88-81ea-57c1f12e97d4","Type":"ContainerDied","Data":"f2681184fb060b1c886c8733d5cf10d9c28ccad75a5a47ab6128b5cfae594567"} Nov 02 21:34:26.546757 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.546732 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:26.630963 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.630926 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-openshift-ca\") pod \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " Nov 02 21:34:26.631172 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.630995 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d036e07-e65e-4b88-81ea-57c1f12e97d4-kube-api-access-9gdbf\" (UniqueName: \"kubernetes.io/projected/7d036e07-e65e-4b88-81ea-57c1f12e97d4-kube-api-access-9gdbf\") pod \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " Nov 02 21:34:26.631172 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.631099 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-xks-ca\") pod \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\" (UID: \"7d036e07-e65e-4b88-81ea-57c1f12e97d4\") " Nov 02 21:34:26.631404 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.631372 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-openshift-ca" pod "7d036e07-e65e-4b88-81ea-57c1f12e97d4" (UID: "7d036e07-e65e-4b88-81ea-57c1f12e97d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:34:26.633198 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.633169 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d036e07-e65e-4b88-81ea-57c1f12e97d4-kube-api-access-9gdbf" pod "7d036e07-e65e-4b88-81ea-57c1f12e97d4" (UID: "7d036e07-e65e-4b88-81ea-57c1f12e97d4"). InnerVolumeSpecName "kube-api-access-9gdbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:34:26.633313 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.633275 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-xks-ca" pod "7d036e07-e65e-4b88-81ea-57c1f12e97d4" (UID: "7d036e07-e65e-4b88-81ea-57c1f12e97d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:34:26.732213 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.732118 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gdbf\" (UniqueName: \"kubernetes.io/projected/7d036e07-e65e-4b88-81ea-57c1f12e97d4-kube-api-access-9gdbf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:26.732213 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.732155 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:26.732213 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:26.732169 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d036e07-e65e-4b88-81ea-57c1f12e97d4-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:34:27.155070 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:27.154952 5800 scope.go:122] "RemoveContainer" containerID="8291f7eb8024ba2ac054022484c5ed21eeb3589c1b37ef35ae32d690a5ac422f" Nov 02 21:34:27.176354 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:27.165134 5800 scope.go:122] "RemoveContainer" containerID="437f2bdd476154d87a36cfb800129c2cf7c2dcf1c1fac15e7ef5c8d120a53aee" Nov 02 21:34:27.407663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:27.407581 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" Nov 02 21:34:27.407663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:27.407616 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr" event={"ID":"7d036e07-e65e-4b88-81ea-57c1f12e97d4","Type":"ContainerDied","Data":"bf9077845f6d05002d70bbd565466c1dc9b888ff98a777b6f28d089df67ed9e7"} Nov 02 21:34:27.407874 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:34:27.407667 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9077845f6d05002d70bbd565466c1dc9b888ff98a777b6f28d089df67ed9e7" Nov 02 21:34:52.694997 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:34:52.694957 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:35:04.879765 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.879682 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5"] Nov 02 21:35:04.880173 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.880160 5800 cpu_manager.go:414] "containerMap: removing container" podUID="7d036e07-e65e-4b88-81ea-57c1f12e97d4" containerName="revoke-keys" Nov 02 21:35:04.880224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.880175 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d036e07-e65e-4b88-81ea-57c1f12e97d4" containerName="revoke-keys" Nov 02 21:35:04.880270 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.880261 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d036e07-e65e-4b88-81ea-57c1f12e97d4" containerName="revoke-keys" Nov 02 21:35:04.883066 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.883049 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:04.934782 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.934728 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5"] Nov 02 21:35:04.977923 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.977882 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:35:04.978130 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.977970 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:35:04.978130 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.977984 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:35:04.980889 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:04.980869 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:35:05.054594 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.054551 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.054770 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.054671 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dsdm\" (UniqueName: \"kubernetes.io/projected/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-kube-api-access-2dsdm\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.054815 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.054771 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.155763 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.155665 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.155925 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.155835 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.155925 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.155907 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dsdm\" (UniqueName: \"kubernetes.io/projected/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-kube-api-access-2dsdm\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.156653 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.156624 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.158204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.158178 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.208920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.208889 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dsdm\" (UniqueName: \"kubernetes.io/projected/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-kube-api-access-2dsdm\") pod \"maas-api-revoke-sub-e2fc9c5b-lxln5\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.497042 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.496960 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:05.863946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:05.863916 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5"] Nov 02 21:35:05.865436 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:35:05.865407 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c665a0_8d46_47d3_b6ee_fca7f38efa9e.slice/crio-439d29b34d8695c0653a7e8273d2176f6c95cfa1701184473b4494017c636347 WatchSource:0}: Error finding container 439d29b34d8695c0653a7e8273d2176f6c95cfa1701184473b4494017c636347: Status 404 returned error can't find the container with id 439d29b34d8695c0653a7e8273d2176f6c95cfa1701184473b4494017c636347 Nov 02 21:35:06.664240 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:06.664208 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" event={"ID":"62c665a0-8d46-47d3-b6ee-fca7f38efa9e","Type":"ContainerStarted","Data":"471a2c85003e2879581801ed83d0e77b8fc9bb80b1f63b14083159030f43284a"} Nov 02 21:35:06.664662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:06.664261 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" event={"ID":"62c665a0-8d46-47d3-b6ee-fca7f38efa9e","Type":"ContainerStarted","Data":"439d29b34d8695c0653a7e8273d2176f6c95cfa1701184473b4494017c636347"} Nov 02 21:35:06.771112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:06.771055 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" podStartSLOduration=2.771041647 podStartE2EDuration="2.771041647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:06.76938289 +0000 UTC m=+6405.286155482" watchObservedRunningTime="2026-11-02 21:35:06.771041647 +0000 UTC m=+6405.287814231" Nov 02 21:35:07.672233 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:07.672196 5800 generic.go:356] "Generic (PLEG): container finished" podID="62c665a0-8d46-47d3-b6ee-fca7f38efa9e" containerID="471a2c85003e2879581801ed83d0e77b8fc9bb80b1f63b14083159030f43284a" exitCode=0 Nov 02 21:35:07.672664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:07.672293 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" event={"ID":"62c665a0-8d46-47d3-b6ee-fca7f38efa9e","Type":"ContainerDied","Data":"471a2c85003e2879581801ed83d0e77b8fc9bb80b1f63b14083159030f43284a"} Nov 02 21:35:08.817697 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:08.817672 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:08.993825 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:08.993741 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-openshift-ca\") pod \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " Nov 02 21:35:08.993825 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:08.993822 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-xks-ca\") pod \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " Nov 02 21:35:08.994098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:08.993875 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-kube-api-access-2dsdm\" (UniqueName: \"kubernetes.io/projected/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-kube-api-access-2dsdm\") pod \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\" (UID: \"62c665a0-8d46-47d3-b6ee-fca7f38efa9e\") " Nov 02 21:35:08.994211 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:08.994183 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-openshift-ca" pod "62c665a0-8d46-47d3-b6ee-fca7f38efa9e" (UID: "62c665a0-8d46-47d3-b6ee-fca7f38efa9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:08.995997 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:08.995975 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-xks-ca" pod "62c665a0-8d46-47d3-b6ee-fca7f38efa9e" (UID: "62c665a0-8d46-47d3-b6ee-fca7f38efa9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:08.996078 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:08.996060 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-kube-api-access-2dsdm" pod "62c665a0-8d46-47d3-b6ee-fca7f38efa9e" (UID: "62c665a0-8d46-47d3-b6ee-fca7f38efa9e"). InnerVolumeSpecName "kube-api-access-2dsdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:09.095195 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:09.095154 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:09.095195 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:09.095191 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:09.095195 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:09.095200 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dsdm\" (UniqueName: \"kubernetes.io/projected/62c665a0-8d46-47d3-b6ee-fca7f38efa9e-kube-api-access-2dsdm\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:09.685104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:09.685070 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" Nov 02 21:35:09.685312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:09.685108 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5" event={"ID":"62c665a0-8d46-47d3-b6ee-fca7f38efa9e","Type":"ContainerDied","Data":"439d29b34d8695c0653a7e8273d2176f6c95cfa1701184473b4494017c636347"} Nov 02 21:35:09.685312 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:09.685180 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="439d29b34d8695c0653a7e8273d2176f6c95cfa1701184473b4494017c636347" Nov 02 21:35:15.628003 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.627969 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w"] Nov 02 21:35:15.628443 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.628429 5800 cpu_manager.go:414] "containerMap: removing container" podUID="62c665a0-8d46-47d3-b6ee-fca7f38efa9e" containerName="revoke-keys" Nov 02 21:35:15.628495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.628444 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c665a0-8d46-47d3-b6ee-fca7f38efa9e" containerName="revoke-keys" Nov 02 21:35:15.628530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.628521 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c665a0-8d46-47d3-b6ee-fca7f38efa9e" containerName="revoke-keys" Nov 02 21:35:15.631931 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.631913 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.656948 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.656905 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rhg\" (UniqueName: \"kubernetes.io/projected/60a99f8c-9289-49b6-8306-db885a3c91d4-kube-api-access-f2rhg\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.657112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.657082 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.657166 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.657145 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.680932 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.680887 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:35:15.680932 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.680902 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:35:15.727361 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.727331 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:35:15.727361 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.727331 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:35:15.730154 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.730125 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w"] Nov 02 21:35:15.757869 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.757832 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.758076 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.757883 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.758076 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.757930 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rhg\" (UniqueName: \"kubernetes.io/projected/60a99f8c-9289-49b6-8306-db885a3c91d4-kube-api-access-f2rhg\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.758596 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.758574 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.760421 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.760392 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.811616 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.811572 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rhg\" (UniqueName: \"kubernetes.io/projected/60a99f8c-9289-49b6-8306-db885a3c91d4-kube-api-access-f2rhg\") pod \"maas-api-revoke-sub-5a29efb3-whp9w\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:15.945792 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:15.945701 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:16.312876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:16.312845 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w"] Nov 02 21:35:16.314352 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:35:16.314273 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a99f8c_9289_49b6_8306_db885a3c91d4.slice/crio-b6fae6269520d7beca460a365c713a0de875ad77a002ad08939c9c5084003b88 WatchSource:0}: Error finding container b6fae6269520d7beca460a365c713a0de875ad77a002ad08939c9c5084003b88: Status 404 returned error can't find the container with id b6fae6269520d7beca460a365c713a0de875ad77a002ad08939c9c5084003b88 Nov 02 21:35:16.728726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:16.728688 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" event={"ID":"60a99f8c-9289-49b6-8306-db885a3c91d4","Type":"ContainerStarted","Data":"15ee01a2bf2c17760c4ad8864ae34222774a485935a8bad3bfb7cb3fad9ed68c"} Nov 02 21:35:16.729159 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:16.728753 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" event={"ID":"60a99f8c-9289-49b6-8306-db885a3c91d4","Type":"ContainerStarted","Data":"b6fae6269520d7beca460a365c713a0de875ad77a002ad08939c9c5084003b88"} Nov 02 21:35:16.834510 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:16.834449 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" podStartSLOduration=1.8344340099999998 podStartE2EDuration="1.83443401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:16.833465951 +0000 UTC m=+6415.350238552" watchObservedRunningTime="2026-11-02 21:35:16.83443401 +0000 UTC m=+6415.351206594" Nov 02 21:35:17.737187 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:17.737153 5800 generic.go:356] "Generic (PLEG): container finished" podID="60a99f8c-9289-49b6-8306-db885a3c91d4" containerID="15ee01a2bf2c17760c4ad8864ae34222774a485935a8bad3bfb7cb3fad9ed68c" exitCode=0 Nov 02 21:35:17.737722 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:17.737245 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" event={"ID":"60a99f8c-9289-49b6-8306-db885a3c91d4","Type":"ContainerDied","Data":"15ee01a2bf2c17760c4ad8864ae34222774a485935a8bad3bfb7cb3fad9ed68c"} Nov 02 21:35:19.046756 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.046722 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:19.206389 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.206349 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-openshift-ca\") pod \"60a99f8c-9289-49b6-8306-db885a3c91d4\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " Nov 02 21:35:19.206702 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.206678 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a99f8c-9289-49b6-8306-db885a3c91d4-kube-api-access-f2rhg\" (UniqueName: \"kubernetes.io/projected/60a99f8c-9289-49b6-8306-db885a3c91d4-kube-api-access-f2rhg\") pod \"60a99f8c-9289-49b6-8306-db885a3c91d4\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " Nov 02 21:35:19.206870 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.206853 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-xks-ca\") pod \"60a99f8c-9289-49b6-8306-db885a3c91d4\" (UID: \"60a99f8c-9289-49b6-8306-db885a3c91d4\") " Nov 02 21:35:19.207954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.207917 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-openshift-ca" pod "60a99f8c-9289-49b6-8306-db885a3c91d4" (UID: "60a99f8c-9289-49b6-8306-db885a3c91d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:19.210471 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.210434 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-xks-ca" pod "60a99f8c-9289-49b6-8306-db885a3c91d4" (UID: "60a99f8c-9289-49b6-8306-db885a3c91d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:19.210986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.210962 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a99f8c-9289-49b6-8306-db885a3c91d4-kube-api-access-f2rhg" pod "60a99f8c-9289-49b6-8306-db885a3c91d4" (UID: "60a99f8c-9289-49b6-8306-db885a3c91d4"). InnerVolumeSpecName "kube-api-access-f2rhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:19.308150 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.308042 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.308150 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.308070 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2rhg\" (UniqueName: \"kubernetes.io/projected/60a99f8c-9289-49b6-8306-db885a3c91d4-kube-api-access-f2rhg\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.308150 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.308080 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a99f8c-9289-49b6-8306-db885a3c91d4-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:19.762622 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.762581 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" event={"ID":"60a99f8c-9289-49b6-8306-db885a3c91d4","Type":"ContainerDied","Data":"b6fae6269520d7beca460a365c713a0de875ad77a002ad08939c9c5084003b88"} Nov 02 21:35:19.762898 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.762877 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6fae6269520d7beca460a365c713a0de875ad77a002ad08939c9c5084003b88" Nov 02 21:35:19.763091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:19.763075 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w" Nov 02 21:35:22.694706 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:35:22.694667 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:35:28.043136 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:28.043089 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4"] Nov 02 21:35:28.089387 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:28.089345 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-lmmd4"] Nov 02 21:35:29.620560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:29.620524 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740b0973-6474-4ae0-b9bf-c55033a946c9" path="/var/lib/kubelet/pods/740b0973-6474-4ae0-b9bf-c55033a946c9/volumes" Nov 02 21:35:34.095849 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.095815 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr"] Nov 02 21:35:34.141837 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.141804 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6wnkr"] Nov 02 21:35:34.209072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.209038 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv"] Nov 02 21:35:34.209706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.209690 5800 cpu_manager.go:414] "containerMap: removing container" podUID="60a99f8c-9289-49b6-8306-db885a3c91d4" containerName="revoke-keys" Nov 02 21:35:34.209762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.209709 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a99f8c-9289-49b6-8306-db885a3c91d4" containerName="revoke-keys" Nov 02 21:35:34.209814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.209804 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a99f8c-9289-49b6-8306-db885a3c91d4" containerName="revoke-keys" Nov 02 21:35:34.213706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.213689 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.264732 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.264684 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:35:34.264948 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.264696 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:35:34.265047 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.264947 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:35:34.265184 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.265054 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:35:34.268488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.268453 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv"] Nov 02 21:35:34.280555 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.280526 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.280722 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.280570 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.280722 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.280668 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qml7\" (UniqueName: \"kubernetes.io/projected/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-kube-api-access-6qml7\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.382327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.382214 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.382327 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.382271 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.382546 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.382340 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qml7\" (UniqueName: \"kubernetes.io/projected/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-kube-api-access-6qml7\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.382946 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.382914 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.384918 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.384892 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.435440 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.435399 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qml7\" (UniqueName: \"kubernetes.io/projected/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-kube-api-access-6qml7\") pod \"maas-api-revoke-sub-3d4156f1-272gv\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.526718 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.526682 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:34.898010 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:34.897976 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv"] Nov 02 21:35:34.899142 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:35:34.899110 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655b1e29_ffcf_4d77_ab30_0cdcec50ddd3.slice/crio-230aa082318967cd78aa05a1901ae5ec7281702d9cdac53000f3b88c84f3b7f6 WatchSource:0}: Error finding container 230aa082318967cd78aa05a1901ae5ec7281702d9cdac53000f3b88c84f3b7f6: Status 404 returned error can't find the container with id 230aa082318967cd78aa05a1901ae5ec7281702d9cdac53000f3b88c84f3b7f6 Nov 02 21:35:35.621541 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:35.621499 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d036e07-e65e-4b88-81ea-57c1f12e97d4" path="/var/lib/kubelet/pods/7d036e07-e65e-4b88-81ea-57c1f12e97d4/volumes" Nov 02 21:35:35.864392 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:35.864351 5800 generic.go:356] "Generic (PLEG): container finished" podID="655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" containerID="ade53c02a5815cd02e3908e46f0978c72cdfe9bca8ba283f001c0b1c305bbdec" exitCode=0 Nov 02 21:35:35.864562 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:35.864435 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" event={"ID":"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3","Type":"ContainerDied","Data":"ade53c02a5815cd02e3908e46f0978c72cdfe9bca8ba283f001c0b1c305bbdec"} Nov 02 21:35:35.864562 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:35.864487 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" event={"ID":"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3","Type":"ContainerStarted","Data":"230aa082318967cd78aa05a1901ae5ec7281702d9cdac53000f3b88c84f3b7f6"} Nov 02 21:35:37.010048 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.010019 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:37.110040 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.110000 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-openshift-ca\") pod \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " Nov 02 21:35:37.110233 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.110122 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-kube-api-access-6qml7\" (UniqueName: \"kubernetes.io/projected/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-kube-api-access-6qml7\") pod \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " Nov 02 21:35:37.110317 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.110268 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-xks-ca\") pod \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\" (UID: \"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3\") " Nov 02 21:35:37.110497 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.110472 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-openshift-ca" pod "655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" (UID: "655b1e29-ffcf-4d77-ab30-0cdcec50ddd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:35:37.110644 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.110629 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:37.112252 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.112227 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-xks-ca" pod "655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" (UID: "655b1e29-ffcf-4d77-ab30-0cdcec50ddd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:35:37.112369 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.112347 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-kube-api-access-6qml7" pod "655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" (UID: "655b1e29-ffcf-4d77-ab30-0cdcec50ddd3"). InnerVolumeSpecName "kube-api-access-6qml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:35:37.211579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.211484 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:37.211579 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.211523 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qml7\" (UniqueName: \"kubernetes.io/projected/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3-kube-api-access-6qml7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:35:37.877407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.877377 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" Nov 02 21:35:37.877407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.877397 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv" event={"ID":"655b1e29-ffcf-4d77-ab30-0cdcec50ddd3","Type":"ContainerDied","Data":"230aa082318967cd78aa05a1901ae5ec7281702d9cdac53000f3b88c84f3b7f6"} Nov 02 21:35:37.877655 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:37.877445 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="230aa082318967cd78aa05a1901ae5ec7281702d9cdac53000f3b88c84f3b7f6" Nov 02 21:35:52.695933 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:35:52.695892 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:35:59.043872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:59.043831 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9"] Nov 02 21:35:59.092547 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:59.092507 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mdmp9"] Nov 02 21:35:59.620666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:35:59.620627 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dece7583-0afb-46b4-9371-21194080e8af" path="/var/lib/kubelet/pods/dece7583-0afb-46b4-9371-21194080e8af/volumes" Nov 02 21:36:22.696429 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:36:22.696388 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:36:23.043870 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:36:23.043828 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8"] Nov 02 21:36:23.089986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:36:23.089946 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-t2bt8"] Nov 02 21:36:23.620657 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:36:23.620622 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1bfec8-6940-4da4-8602-2cb690c52b69" path="/var/lib/kubelet/pods/cb1bfec8-6940-4da4-8602-2cb690c52b69/volumes" Nov 02 21:36:27.200209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:36:27.200172 5800 scope.go:122] "RemoveContainer" containerID="4912bbe925d203d2b7b6146e11c86a3f409b260f8ff625c9006143d4a7cea911" Nov 02 21:36:35.043902 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:36:35.043811 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9"] Nov 02 21:36:35.090043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:36:35.090009 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9jhw9"] Nov 02 21:36:35.622223 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:36:35.622187 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a" path="/var/lib/kubelet/pods/35c21b1f-0e6e-4726-8f08-41d1a9bc7e3a/volumes" Nov 02 21:36:52.695042 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:36:52.694995 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:37:09.864351 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.864312 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w"] Nov 02 21:37:09.865314 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.865258 5800 cpu_manager.go:414] "containerMap: removing container" podUID="655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" containerName="revoke-keys" Nov 02 21:37:09.865513 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.865495 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" containerName="revoke-keys" Nov 02 21:37:09.865955 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.865938 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" containerName="revoke-keys" Nov 02 21:37:09.870742 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.870715 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:09.919995 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.919950 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w"] Nov 02 21:37:09.941504 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.941466 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:09.941670 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.941528 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6fd7\" (UniqueName: \"kubernetes.io/projected/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-kube-api-access-j6fd7\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:09.941670 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.941633 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:09.963494 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.963465 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:37:09.963721 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.963701 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:37:09.966118 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.966098 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:37:09.966358 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:09.966341 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:37:10.042957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.042911 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6fd7\" (UniqueName: \"kubernetes.io/projected/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-kube-api-access-j6fd7\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:10.043152 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.043002 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:10.043152 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.043107 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:10.043360 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.043335 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr"] Nov 02 21:37:10.043922 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.043899 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:10.045523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.045500 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-r6hdr"] Nov 02 21:37:10.045664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.045504 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:10.096702 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.096672 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6fd7\" (UniqueName: \"kubernetes.io/projected/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-kube-api-access-j6fd7\") pod \"maas-api-revoke-sub-f4fcacd0-jxx6w\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:10.185444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.185360 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:10.555109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:10.555078 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w"] Nov 02 21:37:10.556255 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:37:10.556222 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3bc0fc_fce7_4fe0_8513_78e5e3265c9d.slice/crio-09b74de2ef54a75c375fc76fa6986eb162d0f3e56c7b10ad4b7da82c4eb16ada WatchSource:0}: Error finding container 09b74de2ef54a75c375fc76fa6986eb162d0f3e56c7b10ad4b7da82c4eb16ada: Status 404 returned error can't find the container with id 09b74de2ef54a75c375fc76fa6986eb162d0f3e56c7b10ad4b7da82c4eb16ada Nov 02 21:37:11.450482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:11.450443 5800 generic.go:356] "Generic (PLEG): container finished" podID="ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" containerID="c9e2bc09d9852d6e723a3de3e41df2063d72edf9c0212b8538c9285bda9fc2e2" exitCode=0 Nov 02 21:37:11.450876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:11.450520 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" event={"ID":"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d","Type":"ContainerDied","Data":"c9e2bc09d9852d6e723a3de3e41df2063d72edf9c0212b8538c9285bda9fc2e2"} Nov 02 21:37:11.450876 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:11.450570 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" event={"ID":"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d","Type":"ContainerStarted","Data":"09b74de2ef54a75c375fc76fa6986eb162d0f3e56c7b10ad4b7da82c4eb16ada"} Nov 02 21:37:11.622057 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:11.622020 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f99808-7f19-4bb8-a8f0-eb87053fc4ad" path="/var/lib/kubelet/pods/92f99808-7f19-4bb8-a8f0-eb87053fc4ad/volumes" Nov 02 21:37:12.600290 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.600254 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:12.668343 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.668275 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-openshift-ca\") pod \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " Nov 02 21:37:12.668548 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.668414 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-xks-ca\") pod \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " Nov 02 21:37:12.668548 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.668477 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-kube-api-access-j6fd7\" (UniqueName: \"kubernetes.io/projected/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-kube-api-access-j6fd7\") pod \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\" (UID: \"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d\") " Nov 02 21:37:12.668684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.668650 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-openshift-ca" pod "ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" (UID: "ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:37:12.668971 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.668807 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:12.670687 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.670653 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-xks-ca" pod "ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" (UID: "ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:37:12.670802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.670708 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-kube-api-access-j6fd7" pod "ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" (UID: "ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d"). InnerVolumeSpecName "kube-api-access-j6fd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:37:12.769882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.769849 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:12.769882 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:12.769875 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6fd7\" (UniqueName: \"kubernetes.io/projected/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d-kube-api-access-j6fd7\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:37:13.464594 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:13.464567 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" Nov 02 21:37:13.464802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:13.464566 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w" event={"ID":"ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d","Type":"ContainerDied","Data":"09b74de2ef54a75c375fc76fa6986eb162d0f3e56c7b10ad4b7da82c4eb16ada"} Nov 02 21:37:13.464802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:13.464708 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b74de2ef54a75c375fc76fa6986eb162d0f3e56c7b10ad4b7da82c4eb16ada" Nov 02 21:37:21.043113 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:21.043075 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq"] Nov 02 21:37:21.089226 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:21.089188 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-crjkq"] Nov 02 21:37:21.620643 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:21.620607 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400cb5d5-d854-41f7-9b13-52ed357fa189" path="/var/lib/kubelet/pods/400cb5d5-d854-41f7-9b13-52ed357fa189/volumes" Nov 02 21:37:22.696337 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:37:22.696269 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:37:27.237359 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:27.237319 5800 scope.go:122] "RemoveContainer" containerID="d58b3702786b34eafb9f896fc4b9b0ec4b0fbf1711ecf2042be908b755b39980" Nov 02 21:37:27.247570 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:27.247547 5800 scope.go:122] "RemoveContainer" containerID="f69f56eeb51511f8a324c8248d331f56b3197510ce9da0ed3f52b56105649a4b" Nov 02 21:37:52.695986 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:37:52.695947 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:37:55.047757 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:55.047705 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68"] Nov 02 21:37:55.089518 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:55.089480 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2wj68"] Nov 02 21:37:55.621802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:37:55.621767 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea56e9d5-87a4-4d75-987a-0074a573bebf" path="/var/lib/kubelet/pods/ea56e9d5-87a4-4d75-987a-0074a573bebf/volumes" Nov 02 21:38:07.106479 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.106384 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv"] Nov 02 21:38:07.152340 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.152302 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-272gv"] Nov 02 21:38:07.219245 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.219213 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw"] Nov 02 21:38:07.219762 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.219743 5800 cpu_manager.go:414] "containerMap: removing container" podUID="ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" containerName="revoke-keys" Nov 02 21:38:07.219819 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.219763 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" containerName="revoke-keys" Nov 02 21:38:07.219864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.219845 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" containerName="revoke-keys" Nov 02 21:38:07.223273 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.223254 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.273444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.273415 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:38:07.273646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.273416 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:38:07.273646 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.273511 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:38:07.273792 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.273683 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:38:07.276768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.276736 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw"] Nov 02 21:38:07.343484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.343446 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.343661 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.343503 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxhnt\" (UniqueName: \"kubernetes.io/projected/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-kube-api-access-rxhnt\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.343704 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.343676 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.444827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.444735 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.444827 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.444788 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxhnt\" (UniqueName: \"kubernetes.io/projected/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-kube-api-access-rxhnt\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.445090 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.444908 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.445476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.445453 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.447318 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.447296 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.498114 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.498065 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxhnt\" (UniqueName: \"kubernetes.io/projected/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-kube-api-access-rxhnt\") pod \"maas-api-revoke-sub-3d4156f1-jbjcw\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.537007 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.536963 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:07.622209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.622174 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655b1e29-ffcf-4d77-ab30-0cdcec50ddd3" path="/var/lib/kubelet/pods/655b1e29-ffcf-4d77-ab30-0cdcec50ddd3/volumes" Nov 02 21:38:07.903900 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:07.903871 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw"] Nov 02 21:38:07.906749 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:38:07.906720 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cb17d1_1814_458b_89c4_61fc8d4f2c29.slice/crio-8093c9bdb09af73cabc4a69d6900ab842666984f58a547270c45e7dc31ac1d4b WatchSource:0}: Error finding container 8093c9bdb09af73cabc4a69d6900ab842666984f58a547270c45e7dc31ac1d4b: Status 404 returned error can't find the container with id 8093c9bdb09af73cabc4a69d6900ab842666984f58a547270c45e7dc31ac1d4b Nov 02 21:38:08.792452 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:08.792417 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" event={"ID":"e9cb17d1-1814-458b-89c4-61fc8d4f2c29","Type":"ContainerStarted","Data":"e9e7be7cf18331a7054e61d8947623fee1dee3ae90e4fbe8fe1c188d31c66c4c"} Nov 02 21:38:08.792878 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:08.792480 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" event={"ID":"e9cb17d1-1814-458b-89c4-61fc8d4f2c29","Type":"ContainerStarted","Data":"8093c9bdb09af73cabc4a69d6900ab842666984f58a547270c45e7dc31ac1d4b"} Nov 02 21:38:08.900342 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:08.900259 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" podStartSLOduration=1.9002458 podStartE2EDuration="1.9002458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:38:08.898306436 +0000 UTC m=+6587.415079019" watchObservedRunningTime="2026-11-02 21:38:08.9002458 +0000 UTC m=+6587.417018376" Nov 02 21:38:09.800209 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:09.800175 5800 generic.go:356] "Generic (PLEG): container finished" podID="e9cb17d1-1814-458b-89c4-61fc8d4f2c29" containerID="e9e7be7cf18331a7054e61d8947623fee1dee3ae90e4fbe8fe1c188d31c66c4c" exitCode=0 Nov 02 21:38:09.800639 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:09.800266 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" event={"ID":"e9cb17d1-1814-458b-89c4-61fc8d4f2c29","Type":"ContainerDied","Data":"e9e7be7cf18331a7054e61d8947623fee1dee3ae90e4fbe8fe1c188d31c66c4c"} Nov 02 21:38:10.948731 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:10.948701 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:10.980624 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:10.980583 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-xks-ca\") pod \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " Nov 02 21:38:10.980815 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:10.980721 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-kube-api-access-rxhnt\" (UniqueName: \"kubernetes.io/projected/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-kube-api-access-rxhnt\") pod \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " Nov 02 21:38:10.980815 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:10.980772 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-openshift-ca\") pod \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\" (UID: \"e9cb17d1-1814-458b-89c4-61fc8d4f2c29\") " Nov 02 21:38:10.981197 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:10.981167 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-openshift-ca" pod "e9cb17d1-1814-458b-89c4-61fc8d4f2c29" (UID: "e9cb17d1-1814-458b-89c4-61fc8d4f2c29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:10.982838 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:10.982815 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-kube-api-access-rxhnt" pod "e9cb17d1-1814-458b-89c4-61fc8d4f2c29" (UID: "e9cb17d1-1814-458b-89c4-61fc8d4f2c29"). InnerVolumeSpecName "kube-api-access-rxhnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:10.983011 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:10.982993 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-xks-ca" pod "e9cb17d1-1814-458b-89c4-61fc8d4f2c29" (UID: "e9cb17d1-1814-458b-89c4-61fc8d4f2c29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:38:11.082467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:11.082379 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.082467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:11.082411 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxhnt\" (UniqueName: \"kubernetes.io/projected/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-kube-api-access-rxhnt\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.082467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:11.082422 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cb17d1-1814-458b-89c4-61fc8d4f2c29-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.814555 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:11.814512 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" event={"ID":"e9cb17d1-1814-458b-89c4-61fc8d4f2c29","Type":"ContainerDied","Data":"8093c9bdb09af73cabc4a69d6900ab842666984f58a547270c45e7dc31ac1d4b"} Nov 02 21:38:11.814555 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:11.814546 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw" Nov 02 21:38:11.814770 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:11.814578 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8093c9bdb09af73cabc4a69d6900ab842666984f58a547270c45e7dc31ac1d4b" Nov 02 21:38:14.048753 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:14.048716 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9"] Nov 02 21:38:14.094707 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:14.094671 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6qnc9"] Nov 02 21:38:15.620545 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:15.620506 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d961c60e-859d-42f3-8cf6-c1dff0cd5e07" path="/var/lib/kubelet/pods/d961c60e-859d-42f3-8cf6-c1dff0cd5e07/volumes" Nov 02 21:38:22.837048 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:38:22.837006 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:38:23.021442 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:23.021409 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:38:23.054307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:23.054201 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:38:23.321461 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:23.321418 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:38:23.321664 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:23.321484 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:38:23.790306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:23.789402 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:38:23.790306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:23.789482 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:38:25.809060 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:25.809020 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k"] Nov 02 21:38:25.809520 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:25.809232 5800 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" podUID="19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" containerName="manager" containerID="cri-o://0e4656216550d4af0ca6513f65de98680dc0f0fe58434c33fe122ee6a691e5d4" gracePeriod=10 Nov 02 21:38:26.942993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:26.942960 5800 generic.go:356] "Generic (PLEG): container finished" podID="19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" containerID="0e4656216550d4af0ca6513f65de98680dc0f0fe58434c33fe122ee6a691e5d4" exitCode=0 Nov 02 21:38:26.943397 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:26.943038 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" event={"ID":"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b","Type":"ContainerDied","Data":"0e4656216550d4af0ca6513f65de98680dc0f0fe58434c33fe122ee6a691e5d4"} Nov 02 21:38:26.979962 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:26.979934 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 21:38:27.085520 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.085441 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-extensions-socket-volume\") pod \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " Nov 02 21:38:27.085680 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.085533 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-kube-api-access-sdxpn\" (UniqueName: \"kubernetes.io/projected/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-kube-api-access-sdxpn\") pod \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\" (UID: \"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b\") " Nov 02 21:38:27.085870 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.085845 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-extensions-socket-volume" pod "19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" (UID: "19a2f253-46f8-4c5f-ad82-dcd0bac32f1b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:27.087702 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.087676 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-kube-api-access-sdxpn" pod "19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" (UID: "19a2f253-46f8-4c5f-ad82-dcd0bac32f1b"). InnerVolumeSpecName "kube-api-access-sdxpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:27.187092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.187037 5800 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-extensions-socket-volume\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:27.187092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.187077 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxpn\" (UniqueName: \"kubernetes.io/projected/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b-kube-api-access-sdxpn\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:27.279514 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.279478 5800 scope.go:122] "RemoveContainer" containerID="3f0ebdf1ea6406daad24588902baec03186b67a277303c361f11d9be81f8a116" Nov 02 21:38:27.292350 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.292323 5800 scope.go:122] "RemoveContainer" containerID="92417f11885178e68ae46de35426910bb30596d7c8de2215356a14928dccdbac" Nov 02 21:38:27.301343 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.301320 5800 scope.go:122] "RemoveContainer" containerID="5e91784e08674c7c26c60a96359611944d54621bde72654a65822ffdc80855e3" Nov 02 21:38:27.309832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.309809 5800 scope.go:122] "RemoveContainer" containerID="0e4656216550d4af0ca6513f65de98680dc0f0fe58434c33fe122ee6a691e5d4" Nov 02 21:38:27.948788 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.948697 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" event={"ID":"19a2f253-46f8-4c5f-ad82-dcd0bac32f1b","Type":"ContainerDied","Data":"363b86718a3d894ecea6d690a503f97ae3606047b412fc58bff7935103676cfd"} Nov 02 21:38:27.948788 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:27.948775 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k" Nov 02 21:38:28.209413 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:28.209302 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k"] Nov 02 21:38:28.212745 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:28.212711 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-vnq4k"] Nov 02 21:38:29.620668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:29.620632 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" path="/var/lib/kubelet/pods/19a2f253-46f8-4c5f-ad82-dcd0bac32f1b/volumes" Nov 02 21:38:36.043134 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:36.043096 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp"] Nov 02 21:38:36.089563 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:36.089524 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-cxmpp"] Nov 02 21:38:37.620977 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:37.620942 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26398c00-2b3d-46f7-9db7-de7912ab0479" path="/var/lib/kubelet/pods/26398c00-2b3d-46f7-9db7-de7912ab0479/volumes" Nov 02 21:38:52.042956 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:52.042920 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w"] Nov 02 21:38:52.089948 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:52.089917 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hvv5w"] Nov 02 21:38:52.696432 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:38:52.696392 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:38:53.620769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:38:53.620736 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914a01d1-fefe-4d1c-aec9-7414581968fa" path="/var/lib/kubelet/pods/914a01d1-fefe-4d1c-aec9-7414581968fa/volumes" Nov 02 21:39:22.695624 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:39:22.695585 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:39:27.353131 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:27.353091 5800 scope.go:122] "RemoveContainer" containerID="a23d1b77b7a57a2190a674b721d584fa30c4b696187334ab47ddbe1cad03390f" Nov 02 21:39:27.362272 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:27.362249 5800 scope.go:122] "RemoveContainer" containerID="1e16a957d72815f031d9d49e7ffd95e306e274e10393979761be37780b4493b8" Nov 02 21:39:29.308843 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.308805 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt"] Nov 02 21:39:29.309251 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.309204 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e9cb17d1-1814-458b-89c4-61fc8d4f2c29" containerName="revoke-keys" Nov 02 21:39:29.309251 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.309215 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cb17d1-1814-458b-89c4-61fc8d4f2c29" containerName="revoke-keys" Nov 02 21:39:29.309251 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.309249 5800 cpu_manager.go:414] "containerMap: removing container" podUID="19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" containerName="manager" Nov 02 21:39:29.309407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.309254 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" containerName="manager" Nov 02 21:39:29.309407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.309330 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a2f253-46f8-4c5f-ad82-dcd0bac32f1b" containerName="manager" Nov 02 21:39:29.309407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.309344 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cb17d1-1814-458b-89c4-61fc8d4f2c29" containerName="revoke-keys" Nov 02 21:39:29.313773 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.313754 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:29.322598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.322577 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt"] Nov 02 21:39:29.366103 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.366066 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qcrlr\"" Nov 02 21:39:29.414645 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.414600 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7428739-8fc4-4d56-9eeb-8018ac0334f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8wmrt\" (UID: \"c7428739-8fc4-4d56-9eeb-8018ac0334f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:29.414846 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.414792 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5lg\" (UniqueName: \"kubernetes.io/projected/c7428739-8fc4-4d56-9eeb-8018ac0334f3-kube-api-access-bh5lg\") pod \"kuadrant-operator-controller-manager-678d87766f-8wmrt\" (UID: \"c7428739-8fc4-4d56-9eeb-8018ac0334f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:29.516028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.515993 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5lg\" (UniqueName: \"kubernetes.io/projected/c7428739-8fc4-4d56-9eeb-8018ac0334f3-kube-api-access-bh5lg\") pod \"kuadrant-operator-controller-manager-678d87766f-8wmrt\" (UID: \"c7428739-8fc4-4d56-9eeb-8018ac0334f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:29.516255 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.516049 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7428739-8fc4-4d56-9eeb-8018ac0334f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8wmrt\" (UID: \"c7428739-8fc4-4d56-9eeb-8018ac0334f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:29.516437 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.516416 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7428739-8fc4-4d56-9eeb-8018ac0334f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8wmrt\" (UID: \"c7428739-8fc4-4d56-9eeb-8018ac0334f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:29.570274 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.570205 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5lg\" (UniqueName: \"kubernetes.io/projected/c7428739-8fc4-4d56-9eeb-8018ac0334f3-kube-api-access-bh5lg\") pod \"kuadrant-operator-controller-manager-678d87766f-8wmrt\" (UID: \"c7428739-8fc4-4d56-9eeb-8018ac0334f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:29.629747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:29.629711 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:30.093051 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:30.093021 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt"] Nov 02 21:39:30.095354 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:39:30.095326 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7428739_8fc4_4d56_9eeb_8018ac0334f3.slice/crio-ea1dc54544e13ea3d375fc893d04bac1031fd8a0665782a0a242607d92d68cdd WatchSource:0}: Error finding container ea1dc54544e13ea3d375fc893d04bac1031fd8a0665782a0a242607d92d68cdd: Status 404 returned error can't find the container with id ea1dc54544e13ea3d375fc893d04bac1031fd8a0665782a0a242607d92d68cdd Nov 02 21:39:30.335096 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:30.335058 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" event={"ID":"c7428739-8fc4-4d56-9eeb-8018ac0334f3","Type":"ContainerStarted","Data":"c9259c3403dbb3148bc4396d940e0749d0aa30d37c7ae37a78f8487a7aff5d91"} Nov 02 21:39:30.335652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:30.335121 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" event={"ID":"c7428739-8fc4-4d56-9eeb-8018ac0334f3","Type":"ContainerStarted","Data":"ea1dc54544e13ea3d375fc893d04bac1031fd8a0665782a0a242607d92d68cdd"} Nov 02 21:39:30.335652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:30.335158 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:30.446986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:30.446883 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" podStartSLOduration=1.44686819 podStartE2EDuration="1.44686819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:30.445568913 +0000 UTC m=+6668.962341499" watchObservedRunningTime="2026-11-02 21:39:30.44686819 +0000 UTC m=+6668.963640775" Nov 02 21:39:41.344210 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:41.344183 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8wmrt" Nov 02 21:39:52.781309 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:39:52.772012 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:39:53.791557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:53.791504 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:39:53.792422 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:53.792366 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-7h7jm" podUID="48b27c1e-b9de-4d19-9b28-656ee6a0f697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:39:55.820597 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:55.820559 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp"] Nov 02 21:39:55.824062 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:55.824044 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:55.920668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:55.920616 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:39:55.920668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:55.920616 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:39:55.920668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:55.920623 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:39:55.920967 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:55.920699 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:39:55.923384 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:55.923361 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp"] Nov 02 21:39:56.021787 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.021751 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.021991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.021893 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.022113 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.022087 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvtn\" (UniqueName: \"kubernetes.io/projected/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-kube-api-access-hjvtn\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.122806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.122717 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvtn\" (UniqueName: \"kubernetes.io/projected/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-kube-api-access-hjvtn\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.122806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.122765 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.122806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.122804 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.123548 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.123527 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.125316 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.125295 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.176035 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.176000 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvtn\" (UniqueName: \"kubernetes.io/projected/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-kube-api-access-hjvtn\") pod \"maas-api-revoke-sub-d576e68b-799zp\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.437745 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.437662 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:56.807477 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:56.807451 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp"] Nov 02 21:39:56.809250 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:39:56.809218 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bd55cc_1a37_4a9e_94fb_d25db6d6e15b.slice/crio-a51bcbe35e135c54bc7a6eafbfe5f6db69873acd1079ab1bad7e74a4a1545789 WatchSource:0}: Error finding container a51bcbe35e135c54bc7a6eafbfe5f6db69873acd1079ab1bad7e74a4a1545789: Status 404 returned error can't find the container with id a51bcbe35e135c54bc7a6eafbfe5f6db69873acd1079ab1bad7e74a4a1545789 Nov 02 21:39:57.521574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:57.521521 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" event={"ID":"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b","Type":"ContainerStarted","Data":"9600f464ee56908b841c1482ba162e34da625d2338eceaaf2bdebbd2896cf5f6"} Nov 02 21:39:57.521999 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:57.521593 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" event={"ID":"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b","Type":"ContainerStarted","Data":"a51bcbe35e135c54bc7a6eafbfe5f6db69873acd1079ab1bad7e74a4a1545789"} Nov 02 21:39:57.629541 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:57.629482 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" podStartSLOduration=2.629462751 podStartE2EDuration="2.629462751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:57.62805959 +0000 UTC m=+6696.144832188" watchObservedRunningTime="2026-11-02 21:39:57.629462751 +0000 UTC m=+6696.146235333" Nov 02 21:39:58.529588 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:58.529548 5800 generic.go:356] "Generic (PLEG): container finished" podID="e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" containerID="9600f464ee56908b841c1482ba162e34da625d2338eceaaf2bdebbd2896cf5f6" exitCode=0 Nov 02 21:39:58.530061 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:58.529631 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" event={"ID":"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b","Type":"ContainerDied","Data":"9600f464ee56908b841c1482ba162e34da625d2338eceaaf2bdebbd2896cf5f6"} Nov 02 21:39:59.686441 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.686415 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:39:59.757263 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.757225 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-kube-api-access-hjvtn\" (UniqueName: \"kubernetes.io/projected/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-kube-api-access-hjvtn\") pod \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " Nov 02 21:39:59.757470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.757339 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-openshift-ca\") pod \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " Nov 02 21:39:59.757470 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.757414 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-xks-ca\") pod \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\" (UID: \"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b\") " Nov 02 21:39:59.757796 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.757760 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-openshift-ca" pod "e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" (UID: "e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:59.759543 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.759513 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-xks-ca" pod "e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" (UID: "e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:39:59.759677 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.759654 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-kube-api-access-hjvtn" pod "e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" (UID: "e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b"). InnerVolumeSpecName "kube-api-access-hjvtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:59.858957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.858855 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjvtn\" (UniqueName: \"kubernetes.io/projected/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-kube-api-access-hjvtn\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:59.858957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.858894 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:59.858957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:39:59.858912 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:00.542912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:00.542885 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" Nov 02 21:40:00.543071 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:00.542881 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp" event={"ID":"e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b","Type":"ContainerDied","Data":"a51bcbe35e135c54bc7a6eafbfe5f6db69873acd1079ab1bad7e74a4a1545789"} Nov 02 21:40:00.543071 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:00.543017 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a51bcbe35e135c54bc7a6eafbfe5f6db69873acd1079ab1bad7e74a4a1545789" Nov 02 21:40:09.053083 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:09.053041 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5"] Nov 02 21:40:09.094383 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:09.094337 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-lxln5"] Nov 02 21:40:09.620941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:09.620905 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c665a0-8d46-47d3-b6ee-fca7f38efa9e" path="/var/lib/kubelet/pods/62c665a0-8d46-47d3-b6ee-fca7f38efa9e/volumes" Nov 02 21:40:16.904627 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:16.904592 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292"] Nov 02 21:40:16.905023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:16.905006 5800 cpu_manager.go:414] "containerMap: removing container" podUID="e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" containerName="revoke-keys" Nov 02 21:40:16.905023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:16.905017 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" containerName="revoke-keys" Nov 02 21:40:16.905129 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:16.905118 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" containerName="revoke-keys" Nov 02 21:40:16.908235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:16.908218 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:16.959011 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:16.958984 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:40:16.959188 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:16.959131 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:40:17.002035 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.001993 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbm77\" (UniqueName: \"kubernetes.io/projected/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-kube-api-access-pbm77\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.002239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.002116 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.002239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.002190 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.005453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.005436 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:40:17.005513 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.005483 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:40:17.007903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.007886 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292"] Nov 02 21:40:17.103044 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.103009 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.103314 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.103077 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.103314 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.103129 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbm77\" (UniqueName: \"kubernetes.io/projected/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-kube-api-access-pbm77\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.103875 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.103852 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.105604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.105574 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.156371 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.156303 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbm77\" (UniqueName: \"kubernetes.io/projected/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-kube-api-access-pbm77\") pod \"maas-api-revoke-sub-d2bc93bc-2m292\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.221797 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.221749 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:17.586885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.586842 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292"] Nov 02 21:40:17.589648 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:40:17.589615 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663fb314_c00e_4edf_806f_e4c7ccc7cd7e.slice/crio-c371adbbf5d52f65f4562a032bcebfb08ca095d9f7f7ca4b644f6fbf2270d1b0 WatchSource:0}: Error finding container c371adbbf5d52f65f4562a032bcebfb08ca095d9f7f7ca4b644f6fbf2270d1b0: Status 404 returned error can't find the container with id c371adbbf5d52f65f4562a032bcebfb08ca095d9f7f7ca4b644f6fbf2270d1b0 Nov 02 21:40:17.693154 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:17.693115 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" event={"ID":"663fb314-c00e-4edf-806f-e4c7ccc7cd7e","Type":"ContainerStarted","Data":"c371adbbf5d52f65f4562a032bcebfb08ca095d9f7f7ca4b644f6fbf2270d1b0"} Nov 02 21:40:18.700835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:18.700742 5800 generic.go:356] "Generic (PLEG): container finished" podID="663fb314-c00e-4edf-806f-e4c7ccc7cd7e" containerID="3e202d53cca08a781ed93a9b0d79d698705269e59c6aab45ec596da3d797ea0e" exitCode=0 Nov 02 21:40:18.700835 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:18.700791 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" event={"ID":"663fb314-c00e-4edf-806f-e4c7ccc7cd7e","Type":"ContainerDied","Data":"3e202d53cca08a781ed93a9b0d79d698705269e59c6aab45ec596da3d797ea0e"} Nov 02 21:40:19.042599 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:19.042556 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w"] Nov 02 21:40:19.044990 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:19.044961 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-whp9w"] Nov 02 21:40:19.621278 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:19.621243 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a99f8c-9289-49b6-8306-db885a3c91d4" path="/var/lib/kubelet/pods/60a99f8c-9289-49b6-8306-db885a3c91d4/volumes" Nov 02 21:40:19.848671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:19.848641 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:20.032454 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.032418 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-xks-ca\") pod \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " Nov 02 21:40:20.032638 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.032488 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-kube-api-access-pbm77\" (UniqueName: \"kubernetes.io/projected/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-kube-api-access-pbm77\") pod \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " Nov 02 21:40:20.032638 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.032542 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-openshift-ca\") pod \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\" (UID: \"663fb314-c00e-4edf-806f-e4c7ccc7cd7e\") " Nov 02 21:40:20.033035 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.032998 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-openshift-ca" pod "663fb314-c00e-4edf-806f-e4c7ccc7cd7e" (UID: "663fb314-c00e-4edf-806f-e4c7ccc7cd7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:20.034601 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.034569 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-xks-ca" pod "663fb314-c00e-4edf-806f-e4c7ccc7cd7e" (UID: "663fb314-c00e-4edf-806f-e4c7ccc7cd7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:20.034791 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.034771 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-kube-api-access-pbm77" pod "663fb314-c00e-4edf-806f-e4c7ccc7cd7e" (UID: "663fb314-c00e-4edf-806f-e4c7ccc7cd7e"). InnerVolumeSpecName "kube-api-access-pbm77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:20.134125 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.134073 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:20.134125 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.134119 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:20.134125 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.134133 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbm77\" (UniqueName: \"kubernetes.io/projected/663fb314-c00e-4edf-806f-e4c7ccc7cd7e-kube-api-access-pbm77\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:20.714650 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.714616 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" Nov 02 21:40:20.714650 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.714642 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292" event={"ID":"663fb314-c00e-4edf-806f-e4c7ccc7cd7e","Type":"ContainerDied","Data":"c371adbbf5d52f65f4562a032bcebfb08ca095d9f7f7ca4b644f6fbf2270d1b0"} Nov 02 21:40:20.714884 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:20.714686 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c371adbbf5d52f65f4562a032bcebfb08ca095d9f7f7ca4b644f6fbf2270d1b0" Nov 02 21:40:22.696413 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:40:22.696371 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:40:27.389435 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:27.389397 5800 scope.go:122] "RemoveContainer" containerID="f2681184fb060b1c886c8733d5cf10d9c28ccad75a5a47ab6128b5cfae594567" Nov 02 21:40:27.398565 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:27.398546 5800 scope.go:122] "RemoveContainer" containerID="4043db827e984c9e5faa0b9f4059ea7f4aa7cda30cc702a42874b5c668e69ddf" Nov 02 21:40:27.407060 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:27.407039 5800 scope.go:122] "RemoveContainer" containerID="3fac5860f9581e8bd2fdf6ff4935eb197dcdec9f60421ae953dc8c3f9e84b6b6" Nov 02 21:40:27.415495 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:27.415471 5800 scope.go:122] "RemoveContainer" containerID="f7c9e4bf389a34118b3166bf6c819695cf680d802ee3571c28aa36138962638c" Nov 02 21:40:52.711308 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:40:52.710589 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:40:55.400662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.400625 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7"] Nov 02 21:40:55.401079 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.401064 5800 cpu_manager.go:414] "containerMap: removing container" podUID="663fb314-c00e-4edf-806f-e4c7ccc7cd7e" containerName="revoke-keys" Nov 02 21:40:55.401079 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.401078 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663fb314-c00e-4edf-806f-e4c7ccc7cd7e" containerName="revoke-keys" Nov 02 21:40:55.401198 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.401176 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="663fb314-c00e-4edf-806f-e4c7ccc7cd7e" containerName="revoke-keys" Nov 02 21:40:55.404578 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.404558 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.440140 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.440105 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.440371 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.440154 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rv4g\" (UniqueName: \"kubernetes.io/projected/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-kube-api-access-7rv4g\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.440371 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.440258 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.457269 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.457234 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7"] Nov 02 21:40:55.500135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.500088 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:40:55.500135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.500122 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:40:55.500380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.500123 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:40:55.500380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.500158 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:40:55.541438 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.541397 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.541666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.541454 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rv4g\" (UniqueName: \"kubernetes.io/projected/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-kube-api-access-7rv4g\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.541666 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.541505 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.542228 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.542205 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.544092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.544067 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.595126 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.595088 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rv4g\" (UniqueName: \"kubernetes.io/projected/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-kube-api-access-7rv4g\") pod \"maas-api-revoke-sub-738331b1-gdpj7\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:55.717573 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:55.717477 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:56.083663 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:56.083539 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7"] Nov 02 21:40:56.086107 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:40:56.086071 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365129d5_6f90_4ff3_bc9e_a8d7f5efa5a2.slice/crio-d7fbf379c0bd9b09acffdfc18f8746a0cca0a6c0c310ef60817b333501188b41 WatchSource:0}: Error finding container d7fbf379c0bd9b09acffdfc18f8746a0cca0a6c0c310ef60817b333501188b41: Status 404 returned error can't find the container with id d7fbf379c0bd9b09acffdfc18f8746a0cca0a6c0c310ef60817b333501188b41 Nov 02 21:40:56.975204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:56.975166 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" event={"ID":"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2","Type":"ContainerStarted","Data":"fffc9d5a153ea22a76518f18e93d8106fc1f24c728ff4d8beb5360fd401e7303"} Nov 02 21:40:56.975730 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:56.975223 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" event={"ID":"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2","Type":"ContainerStarted","Data":"d7fbf379c0bd9b09acffdfc18f8746a0cca0a6c0c310ef60817b333501188b41"} Nov 02 21:40:57.082805 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:57.082750 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" podStartSLOduration=2.082736628 podStartE2EDuration="2.082736628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:57.080876419 +0000 UTC m=+6755.597649008" watchObservedRunningTime="2026-11-02 21:40:57.082736628 +0000 UTC m=+6755.599509237" Nov 02 21:40:57.983231 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:57.983194 5800 generic.go:356] "Generic (PLEG): container finished" podID="365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" containerID="fffc9d5a153ea22a76518f18e93d8106fc1f24c728ff4d8beb5360fd401e7303" exitCode=0 Nov 02 21:40:57.983752 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:57.983269 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" event={"ID":"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2","Type":"ContainerDied","Data":"fffc9d5a153ea22a76518f18e93d8106fc1f24c728ff4d8beb5360fd401e7303"} Nov 02 21:40:59.126432 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.126405 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:40:59.182104 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.182065 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-openshift-ca\") pod \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " Nov 02 21:40:59.182362 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.182201 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-kube-api-access-7rv4g\" (UniqueName: \"kubernetes.io/projected/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-kube-api-access-7rv4g\") pod \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " Nov 02 21:40:59.182362 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.182249 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-xks-ca\") pod \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\" (UID: \"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2\") " Nov 02 21:40:59.182589 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.182561 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-openshift-ca" pod "365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" (UID: "365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:59.184394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.184365 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-xks-ca" pod "365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" (UID: "365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:59.184537 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.184439 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-kube-api-access-7rv4g" pod "365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" (UID: "365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2"). InnerVolumeSpecName "kube-api-access-7rv4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:59.283806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.283765 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:59.283806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.283799 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rv4g\" (UniqueName: \"kubernetes.io/projected/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-kube-api-access-7rv4g\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:59.283806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.283810 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:59.997467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.997343 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" event={"ID":"365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2","Type":"ContainerDied","Data":"d7fbf379c0bd9b09acffdfc18f8746a0cca0a6c0c310ef60817b333501188b41"} Nov 02 21:40:59.997467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.997390 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7fbf379c0bd9b09acffdfc18f8746a0cca0a6c0c310ef60817b333501188b41" Nov 02 21:40:59.997467 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:40:59.997391 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7" Nov 02 21:41:06.179703 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.179658 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds"] Nov 02 21:41:06.180367 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.180345 5800 cpu_manager.go:414] "containerMap: removing container" podUID="365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" containerName="revoke-keys" Nov 02 21:41:06.180481 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.180368 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" containerName="revoke-keys" Nov 02 21:41:06.180481 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.180475 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" containerName="revoke-keys" Nov 02 21:41:06.183796 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.183773 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.233906 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.233873 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:41:06.256921 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.256885 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.257092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.256979 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dvpx\" (UniqueName: \"kubernetes.io/projected/1ab68440-1897-4f50-86fa-9d2b80002e3c-kube-api-access-8dvpx\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.257138 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.257093 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.279561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.279537 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:41:06.279683 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.279577 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:41:06.279683 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.279622 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:41:06.282006 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.281986 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds"] Nov 02 21:41:06.358676 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.358642 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.358857 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.358721 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.358857 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.358743 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dvpx\" (UniqueName: \"kubernetes.io/projected/1ab68440-1897-4f50-86fa-9d2b80002e3c-kube-api-access-8dvpx\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.359463 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.359434 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.361002 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.360976 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.412622 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.412584 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dvpx\" (UniqueName: \"kubernetes.io/projected/1ab68440-1897-4f50-86fa-9d2b80002e3c-kube-api-access-8dvpx\") pod \"maas-api-revoke-sub-989f229b-kvpds\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.498168 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.498072 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:06.867888 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:06.867865 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds"] Nov 02 21:41:06.869749 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:41:06.869720 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab68440_1897_4f50_86fa_9d2b80002e3c.slice/crio-71ef1cf0fcc83a63c2653ba63953d071ef3aa7d5f91d79ca0eb1ee462777053c WatchSource:0}: Error finding container 71ef1cf0fcc83a63c2653ba63953d071ef3aa7d5f91d79ca0eb1ee462777053c: Status 404 returned error can't find the container with id 71ef1cf0fcc83a63c2653ba63953d071ef3aa7d5f91d79ca0eb1ee462777053c Nov 02 21:41:07.047112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:07.047077 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" event={"ID":"1ab68440-1897-4f50-86fa-9d2b80002e3c","Type":"ContainerStarted","Data":"71ef1cf0fcc83a63c2653ba63953d071ef3aa7d5f91d79ca0eb1ee462777053c"} Nov 02 21:41:08.054051 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:08.054009 5800 generic.go:356] "Generic (PLEG): container finished" podID="1ab68440-1897-4f50-86fa-9d2b80002e3c" containerID="c645df29ee0ca60ceb4db2dd9167230ed7c512cddbc1bcf8d611a50f57724800" exitCode=0 Nov 02 21:41:08.054539 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:08.054094 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" event={"ID":"1ab68440-1897-4f50-86fa-9d2b80002e3c","Type":"ContainerDied","Data":"c645df29ee0ca60ceb4db2dd9167230ed7c512cddbc1bcf8d611a50f57724800"} Nov 02 21:41:09.202901 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.202877 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:09.289332 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.289301 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-xks-ca\") pod \"1ab68440-1897-4f50-86fa-9d2b80002e3c\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " Nov 02 21:41:09.289550 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.289376 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-openshift-ca\") pod \"1ab68440-1897-4f50-86fa-9d2b80002e3c\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " Nov 02 21:41:09.289550 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.289441 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab68440-1897-4f50-86fa-9d2b80002e3c-kube-api-access-8dvpx\" (UniqueName: \"kubernetes.io/projected/1ab68440-1897-4f50-86fa-9d2b80002e3c-kube-api-access-8dvpx\") pod \"1ab68440-1897-4f50-86fa-9d2b80002e3c\" (UID: \"1ab68440-1897-4f50-86fa-9d2b80002e3c\") " Nov 02 21:41:09.289779 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.289750 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-openshift-ca" pod "1ab68440-1897-4f50-86fa-9d2b80002e3c" (UID: "1ab68440-1897-4f50-86fa-9d2b80002e3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:09.291659 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.291633 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-xks-ca" pod "1ab68440-1897-4f50-86fa-9d2b80002e3c" (UID: "1ab68440-1897-4f50-86fa-9d2b80002e3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:41:09.291823 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.291797 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab68440-1897-4f50-86fa-9d2b80002e3c-kube-api-access-8dvpx" pod "1ab68440-1897-4f50-86fa-9d2b80002e3c" (UID: "1ab68440-1897-4f50-86fa-9d2b80002e3c"). InnerVolumeSpecName "kube-api-access-8dvpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:09.390574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.390489 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:09.390574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.390520 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab68440-1897-4f50-86fa-9d2b80002e3c-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:09.390574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:09.390533 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dvpx\" (UniqueName: \"kubernetes.io/projected/1ab68440-1897-4f50-86fa-9d2b80002e3c-kube-api-access-8dvpx\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:10.067398 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:10.067360 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" Nov 02 21:41:10.067599 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:10.067399 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds" event={"ID":"1ab68440-1897-4f50-86fa-9d2b80002e3c","Type":"ContainerDied","Data":"71ef1cf0fcc83a63c2653ba63953d071ef3aa7d5f91d79ca0eb1ee462777053c"} Nov 02 21:41:10.067599 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:10.067444 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ef1cf0fcc83a63c2653ba63953d071ef3aa7d5f91d79ca0eb1ee462777053c" Nov 02 21:41:22.696163 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:41:22.696119 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:41:27.477306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:27.476963 5800 scope.go:122] "RemoveContainer" containerID="15ee01a2bf2c17760c4ad8864ae34222774a485935a8bad3bfb7cb3fad9ed68c" Nov 02 21:41:27.510677 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:27.510648 5800 scope.go:122] "RemoveContainer" containerID="471a2c85003e2879581801ed83d0e77b8fc9bb80b1f63b14083159030f43284a" Nov 02 21:41:39.554717 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.554681 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92"] Nov 02 21:41:39.555202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.555166 5800 cpu_manager.go:414] "containerMap: removing container" podUID="1ab68440-1897-4f50-86fa-9d2b80002e3c" containerName="revoke-keys" Nov 02 21:41:39.555202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.555179 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab68440-1897-4f50-86fa-9d2b80002e3c" containerName="revoke-keys" Nov 02 21:41:39.555299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.555255 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab68440-1897-4f50-86fa-9d2b80002e3c" containerName="revoke-keys" Nov 02 21:41:39.559954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.559931 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.611065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.611011 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92"] Nov 02 21:41:39.621216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.621184 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.621216 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.621219 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.621499 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.621241 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zstd\" (UniqueName: \"kubernetes.io/projected/432fc8ae-575e-408c-a181-a0f39dceb325-kube-api-access-7zstd\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.653548 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.653513 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:41:39.653741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.653600 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:41:39.655336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.655312 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:41:39.655336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.655317 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:41:39.722124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.722088 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.722124 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.722126 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.722407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.722150 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zstd\" (UniqueName: \"kubernetes.io/projected/432fc8ae-575e-408c-a181-a0f39dceb325-kube-api-access-7zstd\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.722883 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.722864 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.724577 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.724554 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.775658 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.775620 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zstd\" (UniqueName: \"kubernetes.io/projected/432fc8ae-575e-408c-a181-a0f39dceb325-kube-api-access-7zstd\") pod \"maas-api-revoke-sub-ea03a972-6dk92\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:39.876919 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:39.876836 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:40.244564 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:40.244540 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92"] Nov 02 21:41:40.246360 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:41:40.246331 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432fc8ae_575e_408c_a181_a0f39dceb325.slice/crio-99d72e7cc0680f50b3c91fb9e3ef1f5165fe8b95ceb5b8c8415d9e7be4714bd2 WatchSource:0}: Error finding container 99d72e7cc0680f50b3c91fb9e3ef1f5165fe8b95ceb5b8c8415d9e7be4714bd2: Status 404 returned error can't find the container with id 99d72e7cc0680f50b3c91fb9e3ef1f5165fe8b95ceb5b8c8415d9e7be4714bd2 Nov 02 21:41:40.259341 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:40.259311 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" event={"ID":"432fc8ae-575e-408c-a181-a0f39dceb325","Type":"ContainerStarted","Data":"99d72e7cc0680f50b3c91fb9e3ef1f5165fe8b95ceb5b8c8415d9e7be4714bd2"} Nov 02 21:41:41.266689 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:41.266655 5800 generic.go:356] "Generic (PLEG): container finished" podID="432fc8ae-575e-408c-a181-a0f39dceb325" containerID="1f78fb4576dcb64819474ac078ad958a93740a6a176cde9989d7f1d68287effd" exitCode=0 Nov 02 21:41:41.267090 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:41.266747 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" event={"ID":"432fc8ae-575e-408c-a181-a0f39dceb325","Type":"ContainerDied","Data":"1f78fb4576dcb64819474ac078ad958a93740a6a176cde9989d7f1d68287effd"} Nov 02 21:41:42.415943 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.415918 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:42.451291 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.451250 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432fc8ae-575e-408c-a181-a0f39dceb325-kube-api-access-7zstd\" (UniqueName: \"kubernetes.io/projected/432fc8ae-575e-408c-a181-a0f39dceb325-kube-api-access-7zstd\") pod \"432fc8ae-575e-408c-a181-a0f39dceb325\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " Nov 02 21:41:42.451484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.451344 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-openshift-ca\") pod \"432fc8ae-575e-408c-a181-a0f39dceb325\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " Nov 02 21:41:42.451484 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.451462 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-xks-ca\") pod \"432fc8ae-575e-408c-a181-a0f39dceb325\" (UID: \"432fc8ae-575e-408c-a181-a0f39dceb325\") " Nov 02 21:41:42.451757 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.451732 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-openshift-ca" pod "432fc8ae-575e-408c-a181-a0f39dceb325" (UID: "432fc8ae-575e-408c-a181-a0f39dceb325"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:42.453511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.453484 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432fc8ae-575e-408c-a181-a0f39dceb325-kube-api-access-7zstd" pod "432fc8ae-575e-408c-a181-a0f39dceb325" (UID: "432fc8ae-575e-408c-a181-a0f39dceb325"). InnerVolumeSpecName "kube-api-access-7zstd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:42.453647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.453578 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-xks-ca" pod "432fc8ae-575e-408c-a181-a0f39dceb325" (UID: "432fc8ae-575e-408c-a181-a0f39dceb325"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:41:42.552992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.552907 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:42.552992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.552938 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zstd\" (UniqueName: \"kubernetes.io/projected/432fc8ae-575e-408c-a181-a0f39dceb325-kube-api-access-7zstd\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:42.552992 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:42.552949 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432fc8ae-575e-408c-a181-a0f39dceb325-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:43.278814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:43.278783 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" Nov 02 21:41:43.278814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:43.278803 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92" event={"ID":"432fc8ae-575e-408c-a181-a0f39dceb325","Type":"ContainerDied","Data":"99d72e7cc0680f50b3c91fb9e3ef1f5165fe8b95ceb5b8c8415d9e7be4714bd2"} Nov 02 21:41:43.279055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:41:43.278856 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d72e7cc0680f50b3c91fb9e3ef1f5165fe8b95ceb5b8c8415d9e7be4714bd2" Nov 02 21:41:52.696533 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:41:52.696491 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:42:06.690307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:06.689387 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-2sf82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:42:06.690307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:06.689475 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-2sf82" podUID="e7d44f90-b9db-4554-9111-75088930d250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:42:13.050020 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:13.049978 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w"] Nov 02 21:42:13.096317 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:13.096270 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4fcacd0-jxx6w"] Nov 02 21:42:13.620496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:13.620449 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d" path="/var/lib/kubelet/pods/ca3bc0fc-fce7-4fe0-8513-78e5e3265c9d/volumes" Nov 02 21:42:21.890349 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.890314 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz"] Nov 02 21:42:21.890834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.890808 5800 cpu_manager.go:414] "containerMap: removing container" podUID="432fc8ae-575e-408c-a181-a0f39dceb325" containerName="revoke-keys" Nov 02 21:42:21.890834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.890821 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432fc8ae-575e-408c-a181-a0f39dceb325" containerName="revoke-keys" Nov 02 21:42:21.890912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.890896 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="432fc8ae-575e-408c-a181-a0f39dceb325" containerName="revoke-keys" Nov 02 21:42:21.894045 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.894028 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:21.943618 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.943585 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:42:21.946530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.946501 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz"] Nov 02 21:42:21.990044 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.990009 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:42:21.990248 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.990046 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:42:21.990248 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:21.990053 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:42:22.005502 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.005469 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhj2\" (UniqueName: \"kubernetes.io/projected/84882581-4345-416a-8076-2bc35fa56aaa-kube-api-access-qjhj2\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.005675 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.005518 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.005675 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.005661 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.106709 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.106669 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhj2\" (UniqueName: \"kubernetes.io/projected/84882581-4345-416a-8076-2bc35fa56aaa-kube-api-access-qjhj2\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.106903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.106720 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.106903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.106760 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.107506 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.107485 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.109035 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.109016 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.159884 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.159809 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhj2\" (UniqueName: \"kubernetes.io/projected/84882581-4345-416a-8076-2bc35fa56aaa-kube-api-access-qjhj2\") pod \"maas-api-revoke-sub-bbcf6212-mc8nz\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.207718 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.207682 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:22.573595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:22.573565 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz"] Nov 02 21:42:22.574417 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:42:22.574392 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84882581_4345_416a_8076_2bc35fa56aaa.slice/crio-66f6d051057770f724ef013fb9c5d76d12b8c541882c6a9b582974820d37159e WatchSource:0}: Error finding container 66f6d051057770f724ef013fb9c5d76d12b8c541882c6a9b582974820d37159e: Status 404 returned error can't find the container with id 66f6d051057770f724ef013fb9c5d76d12b8c541882c6a9b582974820d37159e Nov 02 21:42:22.696224 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:42:22.696186 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:42:23.542340 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:23.542269 5800 generic.go:356] "Generic (PLEG): container finished" podID="84882581-4345-416a-8076-2bc35fa56aaa" containerID="ac844cd6051330d563140da4a6aa33d6551c1a8029593db2740c6366270d1c40" exitCode=0 Nov 02 21:42:23.542741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:23.542312 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" event={"ID":"84882581-4345-416a-8076-2bc35fa56aaa","Type":"ContainerDied","Data":"ac844cd6051330d563140da4a6aa33d6551c1a8029593db2740c6366270d1c40"} Nov 02 21:42:23.542741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:23.542415 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" event={"ID":"84882581-4345-416a-8076-2bc35fa56aaa","Type":"ContainerStarted","Data":"66f6d051057770f724ef013fb9c5d76d12b8c541882c6a9b582974820d37159e"} Nov 02 21:42:24.696486 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.696460 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:24.832984 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.832898 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-xks-ca\") pod \"84882581-4345-416a-8076-2bc35fa56aaa\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " Nov 02 21:42:24.833160 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.833122 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84882581-4345-416a-8076-2bc35fa56aaa-kube-api-access-qjhj2\" (UniqueName: \"kubernetes.io/projected/84882581-4345-416a-8076-2bc35fa56aaa-kube-api-access-qjhj2\") pod \"84882581-4345-416a-8076-2bc35fa56aaa\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " Nov 02 21:42:24.833249 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.833155 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-openshift-ca\") pod \"84882581-4345-416a-8076-2bc35fa56aaa\" (UID: \"84882581-4345-416a-8076-2bc35fa56aaa\") " Nov 02 21:42:24.833647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.833612 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-openshift-ca" pod "84882581-4345-416a-8076-2bc35fa56aaa" (UID: "84882581-4345-416a-8076-2bc35fa56aaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:42:24.835091 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.835067 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-xks-ca" pod "84882581-4345-416a-8076-2bc35fa56aaa" (UID: "84882581-4345-416a-8076-2bc35fa56aaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:42:24.835346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.835330 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84882581-4345-416a-8076-2bc35fa56aaa-kube-api-access-qjhj2" pod "84882581-4345-416a-8076-2bc35fa56aaa" (UID: "84882581-4345-416a-8076-2bc35fa56aaa"). InnerVolumeSpecName "kube-api-access-qjhj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:42:24.934769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.934731 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhj2\" (UniqueName: \"kubernetes.io/projected/84882581-4345-416a-8076-2bc35fa56aaa-kube-api-access-qjhj2\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:24.934769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.934763 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:24.934769 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:24.934773 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84882581-4345-416a-8076-2bc35fa56aaa-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:25.555023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:25.554987 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" Nov 02 21:42:25.555023 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:25.555014 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz" event={"ID":"84882581-4345-416a-8076-2bc35fa56aaa","Type":"ContainerDied","Data":"66f6d051057770f724ef013fb9c5d76d12b8c541882c6a9b582974820d37159e"} Nov 02 21:42:25.555261 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:25.555064 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f6d051057770f724ef013fb9c5d76d12b8c541882c6a9b582974820d37159e" Nov 02 21:42:27.628016 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:27.627983 5800 scope.go:122] "RemoveContainer" containerID="ade53c02a5815cd02e3908e46f0978c72cdfe9bca8ba283f001c0b1c305bbdec" Nov 02 21:42:32.642391 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.642309 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq"] Nov 02 21:42:32.642804 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.642763 5800 cpu_manager.go:414] "containerMap: removing container" podUID="84882581-4345-416a-8076-2bc35fa56aaa" containerName="revoke-keys" Nov 02 21:42:32.642804 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.642777 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84882581-4345-416a-8076-2bc35fa56aaa" containerName="revoke-keys" Nov 02 21:42:32.642891 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.642847 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="84882581-4345-416a-8076-2bc35fa56aaa" containerName="revoke-keys" Nov 02 21:42:32.647348 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.647326 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.698915 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.698872 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq"] Nov 02 21:42:32.706207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.706157 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.706440 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.706414 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsc98\" (UniqueName: \"kubernetes.io/projected/bd029bea-c23a-440f-a179-f08627ce1046-kube-api-access-qsc98\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.706575 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.706552 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.742089 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.742023 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:42:32.742089 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.742031 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:42:32.742089 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.742039 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:42:32.743044 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.743025 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:42:32.808022 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.807984 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.808204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.808043 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.808204 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.808095 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsc98\" (UniqueName: \"kubernetes.io/projected/bd029bea-c23a-440f-a179-f08627ce1046-kube-api-access-qsc98\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.808802 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.808780 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.810411 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.810388 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.861959 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.861920 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsc98\" (UniqueName: \"kubernetes.io/projected/bd029bea-c23a-440f-a179-f08627ce1046-kube-api-access-qsc98\") pod \"maas-api-revoke-sub-db968f7a-gwltq\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:32.961090 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:32.961010 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:33.332917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:33.332887 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq"] Nov 02 21:42:33.335218 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:42:33.335176 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd029bea_c23a_440f_a179_f08627ce1046.slice/crio-a3ed7d502c281936bba6d59efbe1dacf9ccf825b01126c060b2c6ac23c849005 WatchSource:0}: Error finding container a3ed7d502c281936bba6d59efbe1dacf9ccf825b01126c060b2c6ac23c849005: Status 404 returned error can't find the container with id a3ed7d502c281936bba6d59efbe1dacf9ccf825b01126c060b2c6ac23c849005 Nov 02 21:42:33.602323 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:33.602222 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" event={"ID":"bd029bea-c23a-440f-a179-f08627ce1046","Type":"ContainerStarted","Data":"a3ed7d502c281936bba6d59efbe1dacf9ccf825b01126c060b2c6ac23c849005"} Nov 02 21:42:34.610498 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:34.610462 5800 generic.go:356] "Generic (PLEG): container finished" podID="bd029bea-c23a-440f-a179-f08627ce1046" containerID="848c335598c5166289b2ddde1e01b46cfb035d96288e2f1cb001898f8fb6b189" exitCode=0 Nov 02 21:42:34.610927 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:34.610538 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" event={"ID":"bd029bea-c23a-440f-a179-f08627ce1046","Type":"ContainerDied","Data":"848c335598c5166289b2ddde1e01b46cfb035d96288e2f1cb001898f8fb6b189"} Nov 02 21:42:35.767112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.767080 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:35.839939 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.839900 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-openshift-ca\") pod \"bd029bea-c23a-440f-a179-f08627ce1046\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " Nov 02 21:42:35.840141 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.840065 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-xks-ca\") pod \"bd029bea-c23a-440f-a179-f08627ce1046\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " Nov 02 21:42:35.840141 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.840090 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd029bea-c23a-440f-a179-f08627ce1046-kube-api-access-qsc98\" (UniqueName: \"kubernetes.io/projected/bd029bea-c23a-440f-a179-f08627ce1046-kube-api-access-qsc98\") pod \"bd029bea-c23a-440f-a179-f08627ce1046\" (UID: \"bd029bea-c23a-440f-a179-f08627ce1046\") " Nov 02 21:42:35.840394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.840362 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-openshift-ca" pod "bd029bea-c23a-440f-a179-f08627ce1046" (UID: "bd029bea-c23a-440f-a179-f08627ce1046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:42:35.842307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.842269 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-xks-ca" pod "bd029bea-c23a-440f-a179-f08627ce1046" (UID: "bd029bea-c23a-440f-a179-f08627ce1046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:42:35.842428 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.842407 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd029bea-c23a-440f-a179-f08627ce1046-kube-api-access-qsc98" pod "bd029bea-c23a-440f-a179-f08627ce1046" (UID: "bd029bea-c23a-440f-a179-f08627ce1046"). InnerVolumeSpecName "kube-api-access-qsc98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:42:35.941853 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.941766 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:35.941853 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.941796 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsc98\" (UniqueName: \"kubernetes.io/projected/bd029bea-c23a-440f-a179-f08627ce1046-kube-api-access-qsc98\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:35.941853 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:35.941807 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd029bea-c23a-440f-a179-f08627ce1046-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:42:36.624055 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:36.624027 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" Nov 02 21:42:36.624244 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:36.624056 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq" event={"ID":"bd029bea-c23a-440f-a179-f08627ce1046","Type":"ContainerDied","Data":"a3ed7d502c281936bba6d59efbe1dacf9ccf825b01126c060b2c6ac23c849005"} Nov 02 21:42:36.624244 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:42:36.624100 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ed7d502c281936bba6d59efbe1dacf9ccf825b01126c060b2c6ac23c849005" Nov 02 21:42:52.699381 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:42:52.699322 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:43:11.042732 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:11.042691 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw"] Nov 02 21:43:11.089206 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:11.089169 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbjcw"] Nov 02 21:43:11.621001 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:11.620968 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cb17d1-1814-458b-89c4-61fc8d4f2c29" path="/var/lib/kubelet/pods/e9cb17d1-1814-458b-89c4-61fc8d4f2c29/volumes" Nov 02 21:43:17.212881 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.212844 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk"] Nov 02 21:43:17.213307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.213298 5800 cpu_manager.go:414] "containerMap: removing container" podUID="bd029bea-c23a-440f-a179-f08627ce1046" containerName="revoke-keys" Nov 02 21:43:17.213364 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.213310 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd029bea-c23a-440f-a179-f08627ce1046" containerName="revoke-keys" Nov 02 21:43:17.213407 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.213375 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd029bea-c23a-440f-a179-f08627ce1046" containerName="revoke-keys" Nov 02 21:43:17.216545 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.216528 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.305851 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.305811 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.306043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.305882 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.306043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.305999 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbvl\" (UniqueName: \"kubernetes.io/projected/9b5a0964-e15b-4b4a-8163-5a60fae981b7-kube-api-access-nmbvl\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.313604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.313577 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:43:17.313726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.313607 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:43:17.313726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.313625 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:43:17.313726 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.313666 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:43:17.316203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.316183 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk"] Nov 02 21:43:17.407531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.407491 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.407748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.407557 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.407748 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.407613 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbvl\" (UniqueName: \"kubernetes.io/projected/9b5a0964-e15b-4b4a-8163-5a60fae981b7-kube-api-access-nmbvl\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.408207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.408184 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.410053 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.410022 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.461033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.461001 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbvl\" (UniqueName: \"kubernetes.io/projected/9b5a0964-e15b-4b4a-8163-5a60fae981b7-kube-api-access-nmbvl\") pod \"maas-api-revoke-sub-c2db92a0-f88tk\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.529592 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.529559 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:17.894970 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.894939 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk"] Nov 02 21:43:17.896186 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:43:17.896150 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5a0964_e15b_4b4a_8163_5a60fae981b7.slice/crio-10991cd579db8fbe04ec65149e3e91c98c52c9914828739b0cee4031f64404ba WatchSource:0}: Error finding container 10991cd579db8fbe04ec65149e3e91c98c52c9914828739b0cee4031f64404ba: Status 404 returned error can't find the container with id 10991cd579db8fbe04ec65149e3e91c98c52c9914828739b0cee4031f64404ba Nov 02 21:43:17.952227 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:17.952193 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" event={"ID":"9b5a0964-e15b-4b4a-8163-5a60fae981b7","Type":"ContainerStarted","Data":"10991cd579db8fbe04ec65149e3e91c98c52c9914828739b0cee4031f64404ba"} Nov 02 21:43:18.959164 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:18.959130 5800 generic.go:356] "Generic (PLEG): container finished" podID="9b5a0964-e15b-4b4a-8163-5a60fae981b7" containerID="aeeb2762ee8372bcf4f15644829d65e831940f752e37c872b21c8c883f853632" exitCode=0 Nov 02 21:43:18.959585 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:18.959214 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" event={"ID":"9b5a0964-e15b-4b4a-8163-5a60fae981b7","Type":"ContainerDied","Data":"aeeb2762ee8372bcf4f15644829d65e831940f752e37c872b21c8c883f853632"} Nov 02 21:43:20.113177 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.113141 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:20.246243 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.246152 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-xks-ca\") pod \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " Nov 02 21:43:20.246243 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.246229 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-openshift-ca\") pod \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " Nov 02 21:43:20.246523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.246271 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5a0964-e15b-4b4a-8163-5a60fae981b7-kube-api-access-nmbvl\" (UniqueName: \"kubernetes.io/projected/9b5a0964-e15b-4b4a-8163-5a60fae981b7-kube-api-access-nmbvl\") pod \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\" (UID: \"9b5a0964-e15b-4b4a-8163-5a60fae981b7\") " Nov 02 21:43:20.246612 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.246585 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-openshift-ca" pod "9b5a0964-e15b-4b4a-8163-5a60fae981b7" (UID: "9b5a0964-e15b-4b4a-8163-5a60fae981b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:20.248496 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.248465 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5a0964-e15b-4b4a-8163-5a60fae981b7-kube-api-access-nmbvl" pod "9b5a0964-e15b-4b4a-8163-5a60fae981b7" (UID: "9b5a0964-e15b-4b4a-8163-5a60fae981b7"). InnerVolumeSpecName "kube-api-access-nmbvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:20.248626 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.248574 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-xks-ca" pod "9b5a0964-e15b-4b4a-8163-5a60fae981b7" (UID: "9b5a0964-e15b-4b4a-8163-5a60fae981b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:20.347652 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.347610 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:20.347831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.347674 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5a0964-e15b-4b4a-8163-5a60fae981b7-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:20.347831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.347689 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbvl\" (UniqueName: \"kubernetes.io/projected/9b5a0964-e15b-4b4a-8163-5a60fae981b7-kube-api-access-nmbvl\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:20.973185 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.973151 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" Nov 02 21:43:20.973185 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.973176 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2db92a0-f88tk" event={"ID":"9b5a0964-e15b-4b4a-8163-5a60fae981b7","Type":"ContainerDied","Data":"10991cd579db8fbe04ec65149e3e91c98c52c9914828739b0cee4031f64404ba"} Nov 02 21:43:20.973427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:20.973227 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10991cd579db8fbe04ec65149e3e91c98c52c9914828739b0cee4031f64404ba" Nov 02 21:43:22.695840 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:43:22.695788 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:43:23.128681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:23.128644 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:43:23.132383 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:23.132359 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-2b245_deaf3243-1eeb-4d94-bef9-524b67035ba2/console-operator/2.log" Nov 02 21:43:27.651182 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:27.651142 5800 scope.go:122] "RemoveContainer" containerID="c9e2bc09d9852d6e723a3de3e41df2063d72edf9c0212b8538c9285bda9fc2e2" Nov 02 21:43:27.919944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:27.919856 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r"] Nov 02 21:43:27.920308 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:27.920270 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9b5a0964-e15b-4b4a-8163-5a60fae981b7" containerName="revoke-keys" Nov 02 21:43:27.920373 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:27.920309 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5a0964-e15b-4b4a-8163-5a60fae981b7" containerName="revoke-keys" Nov 02 21:43:27.920424 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:27.920414 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5a0964-e15b-4b4a-8163-5a60fae981b7" containerName="revoke-keys" Nov 02 21:43:27.924747 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:27.924721 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:27.973741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:27.973707 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:43:28.020207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.020163 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:43:28.020207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.020170 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:43:28.020207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.020201 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:43:28.023385 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.023354 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r"] Nov 02 21:43:28.027964 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.027933 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.028095 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.028078 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nml7x\" (UniqueName: \"kubernetes.io/projected/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-kube-api-access-nml7x\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.028190 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.028166 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.128894 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.128849 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.129112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.128942 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.129112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.129016 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nml7x\" (UniqueName: \"kubernetes.io/projected/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-kube-api-access-nml7x\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.129557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.129534 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.131538 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.131517 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.182098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.182021 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nml7x\" (UniqueName: \"kubernetes.io/projected/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-kube-api-access-nml7x\") pod \"maas-api-revoke-sub-7678f6d5-fpp9r\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.238525 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.238494 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:28.606986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:28.606961 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r"] Nov 02 21:43:28.608857 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:43:28.608830 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551c87f3_ac6e_4aef_97a7_9651d08f6a9b.slice/crio-f2e50c620991df7c31c0760421229133a57a096b7b6f54899afd31a800f0539e WatchSource:0}: Error finding container f2e50c620991df7c31c0760421229133a57a096b7b6f54899afd31a800f0539e: Status 404 returned error can't find the container with id f2e50c620991df7c31c0760421229133a57a096b7b6f54899afd31a800f0539e Nov 02 21:43:29.024129 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:29.024095 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" event={"ID":"551c87f3-ac6e-4aef-97a7-9651d08f6a9b","Type":"ContainerStarted","Data":"b161d51d463f7b7c4ebea16831867be73ab6e0d93c79d7a22808b3db0dfc65f2"} Nov 02 21:43:29.024626 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:29.024149 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" event={"ID":"551c87f3-ac6e-4aef-97a7-9651d08f6a9b","Type":"ContainerStarted","Data":"f2e50c620991df7c31c0760421229133a57a096b7b6f54899afd31a800f0539e"} Nov 02 21:43:29.131831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:29.131771 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" podStartSLOduration=2.131757372 podStartE2EDuration="2.131757372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:29.129144747 +0000 UTC m=+6907.645917331" watchObservedRunningTime="2026-11-02 21:43:29.131757372 +0000 UTC m=+6907.648529991" Nov 02 21:43:30.032109 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:30.032067 5800 generic.go:356] "Generic (PLEG): container finished" podID="551c87f3-ac6e-4aef-97a7-9651d08f6a9b" containerID="b161d51d463f7b7c4ebea16831867be73ab6e0d93c79d7a22808b3db0dfc65f2" exitCode=0 Nov 02 21:43:30.032687 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:30.032158 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" event={"ID":"551c87f3-ac6e-4aef-97a7-9651d08f6a9b","Type":"ContainerDied","Data":"b161d51d463f7b7c4ebea16831867be73ab6e0d93c79d7a22808b3db0dfc65f2"} Nov 02 21:43:31.399003 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.398920 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:31.498937 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.498886 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-kube-api-access-nml7x\" (UniqueName: \"kubernetes.io/projected/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-kube-api-access-nml7x\") pod \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " Nov 02 21:43:31.500089 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.499748 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-xks-ca\") pod \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " Nov 02 21:43:31.500406 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.500386 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-openshift-ca\") pod \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\" (UID: \"551c87f3-ac6e-4aef-97a7-9651d08f6a9b\") " Nov 02 21:43:31.516034 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.515989 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-openshift-ca" pod "551c87f3-ac6e-4aef-97a7-9651d08f6a9b" (UID: "551c87f3-ac6e-4aef-97a7-9651d08f6a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:31.517303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.517249 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-kube-api-access-nml7x" pod "551c87f3-ac6e-4aef-97a7-9651d08f6a9b" (UID: "551c87f3-ac6e-4aef-97a7-9651d08f6a9b"). InnerVolumeSpecName "kube-api-access-nml7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:31.521420 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.520727 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-xks-ca" pod "551c87f3-ac6e-4aef-97a7-9651d08f6a9b" (UID: "551c87f3-ac6e-4aef-97a7-9651d08f6a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:43:31.618304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.614947 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nml7x\" (UniqueName: \"kubernetes.io/projected/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-kube-api-access-nml7x\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:31.618304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.614989 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:31.618304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:31.615006 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/551c87f3-ac6e-4aef-97a7-9651d08f6a9b-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:32.059305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:32.058845 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" event={"ID":"551c87f3-ac6e-4aef-97a7-9651d08f6a9b","Type":"ContainerDied","Data":"f2e50c620991df7c31c0760421229133a57a096b7b6f54899afd31a800f0539e"} Nov 02 21:43:32.059305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:32.058913 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e50c620991df7c31c0760421229133a57a096b7b6f54899afd31a800f0539e" Nov 02 21:43:32.059305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:43:32.058988 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7678f6d5-fpp9r" Nov 02 21:43:52.697706 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:43:52.697663 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:44:16.010617 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.010508 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv"] Nov 02 21:44:16.011336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.011030 5800 cpu_manager.go:414] "containerMap: removing container" podUID="551c87f3-ac6e-4aef-97a7-9651d08f6a9b" containerName="revoke-keys" Nov 02 21:44:16.011336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.011046 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551c87f3-ac6e-4aef-97a7-9651d08f6a9b" containerName="revoke-keys" Nov 02 21:44:16.011336 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.011122 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="551c87f3-ac6e-4aef-97a7-9651d08f6a9b" containerName="revoke-keys" Nov 02 21:44:16.014695 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.014673 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.066249 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.066205 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv"] Nov 02 21:44:16.108834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.108786 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:44:16.109039 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.108858 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:44:16.109039 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.108993 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:44:16.110440 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.110422 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:44:16.137394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.137341 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkq9\" (UniqueName: \"kubernetes.io/projected/4131915b-6e43-4501-a28f-d80587a53c93-kube-api-access-bgkq9\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.137604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.137418 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.137604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.137514 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.239101 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.239062 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkq9\" (UniqueName: \"kubernetes.io/projected/4131915b-6e43-4501-a28f-d80587a53c93-kube-api-access-bgkq9\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.239369 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.239127 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.239369 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.239181 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.239913 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.239889 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.241842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.241820 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.292841 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.292745 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkq9\" (UniqueName: \"kubernetes.io/projected/4131915b-6e43-4501-a28f-d80587a53c93-kube-api-access-bgkq9\") pod \"maas-api-revoke-sub-10d549e0-dvxrv\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.330033 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.329995 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:16.703893 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:16.703863 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv"] Nov 02 21:44:16.705973 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:44:16.705934 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4131915b_6e43_4501_a28f_d80587a53c93.slice/crio-ea10739bebd5703a95683718276eb929b828665277726445f9ab1feefb38fc3f WatchSource:0}: Error finding container ea10739bebd5703a95683718276eb929b828665277726445f9ab1feefb38fc3f: Status 404 returned error can't find the container with id ea10739bebd5703a95683718276eb929b828665277726445f9ab1feefb38fc3f Nov 02 21:44:17.434028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:17.433867 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" event={"ID":"4131915b-6e43-4501-a28f-d80587a53c93","Type":"ContainerStarted","Data":"b0fdc40f90c6d00be538bd3ce4e4bea370613551aa4bdfa59d459652619ff80d"} Nov 02 21:44:17.434028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:17.433962 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" event={"ID":"4131915b-6e43-4501-a28f-d80587a53c93","Type":"ContainerStarted","Data":"ea10739bebd5703a95683718276eb929b828665277726445f9ab1feefb38fc3f"} Nov 02 21:44:17.541099 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:17.541015 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" podStartSLOduration=2.540992095 podStartE2EDuration="2.540992095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:17.54099035 +0000 UTC m=+6956.057762937" watchObservedRunningTime="2026-11-02 21:44:17.540992095 +0000 UTC m=+6956.057764684" Nov 02 21:44:18.442600 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:18.442539 5800 generic.go:356] "Generic (PLEG): container finished" podID="4131915b-6e43-4501-a28f-d80587a53c93" containerID="b0fdc40f90c6d00be538bd3ce4e4bea370613551aa4bdfa59d459652619ff80d" exitCode=0 Nov 02 21:44:18.443043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:18.442618 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" event={"ID":"4131915b-6e43-4501-a28f-d80587a53c93","Type":"ContainerDied","Data":"b0fdc40f90c6d00be538bd3ce4e4bea370613551aa4bdfa59d459652619ff80d"} Nov 02 21:44:19.598774 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.598746 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:19.677550 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.677509 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-openshift-ca\") pod \"4131915b-6e43-4501-a28f-d80587a53c93\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " Nov 02 21:44:19.677741 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.677636 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-xks-ca\") pod \"4131915b-6e43-4501-a28f-d80587a53c93\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " Nov 02 21:44:19.677789 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.677777 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4131915b-6e43-4501-a28f-d80587a53c93-kube-api-access-bgkq9\" (UniqueName: \"kubernetes.io/projected/4131915b-6e43-4501-a28f-d80587a53c93-kube-api-access-bgkq9\") pod \"4131915b-6e43-4501-a28f-d80587a53c93\" (UID: \"4131915b-6e43-4501-a28f-d80587a53c93\") " Nov 02 21:44:19.677988 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.677962 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-openshift-ca" pod "4131915b-6e43-4501-a28f-d80587a53c93" (UID: "4131915b-6e43-4501-a28f-d80587a53c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:44:19.678143 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.678126 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:19.679830 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.679793 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-xks-ca" pod "4131915b-6e43-4501-a28f-d80587a53c93" (UID: "4131915b-6e43-4501-a28f-d80587a53c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:19.679973 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.679868 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4131915b-6e43-4501-a28f-d80587a53c93-kube-api-access-bgkq9" pod "4131915b-6e43-4501-a28f-d80587a53c93" (UID: "4131915b-6e43-4501-a28f-d80587a53c93"). InnerVolumeSpecName "kube-api-access-bgkq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:19.779558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.779511 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkq9\" (UniqueName: \"kubernetes.io/projected/4131915b-6e43-4501-a28f-d80587a53c93-kube-api-access-bgkq9\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:19.779558 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:19.779552 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4131915b-6e43-4501-a28f-d80587a53c93-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:20.457394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:20.457352 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" Nov 02 21:44:20.457394 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:20.457377 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10d549e0-dvxrv" event={"ID":"4131915b-6e43-4501-a28f-d80587a53c93","Type":"ContainerDied","Data":"ea10739bebd5703a95683718276eb929b828665277726445f9ab1feefb38fc3f"} Nov 02 21:44:20.457638 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:20.457428 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea10739bebd5703a95683718276eb929b828665277726445f9ab1feefb38fc3f" Nov 02 21:44:22.698585 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:44:22.698539 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:44:26.684684 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.684648 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd"] Nov 02 21:44:26.685309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.685124 5800 cpu_manager.go:414] "containerMap: removing container" podUID="4131915b-6e43-4501-a28f-d80587a53c93" containerName="revoke-keys" Nov 02 21:44:26.685309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.685139 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4131915b-6e43-4501-a28f-d80587a53c93" containerName="revoke-keys" Nov 02 21:44:26.685309 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.685246 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="4131915b-6e43-4501-a28f-d80587a53c93" containerName="revoke-keys" Nov 02 21:44:26.688668 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.688644 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.741263 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.741223 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd"] Nov 02 21:44:26.783991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.783951 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:44:26.783991 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.783978 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:44:26.784234 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.783977 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:44:26.784234 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.784124 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:44:26.864701 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.864663 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.864912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.864717 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvst\" (UniqueName: \"kubernetes.io/projected/52c794c5-eabb-4fad-b824-d190a0005ebd-kube-api-access-8nvst\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.864912 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.864894 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.966343 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.966221 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.966343 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.966316 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.966560 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.966353 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvst\" (UniqueName: \"kubernetes.io/projected/52c794c5-eabb-4fad-b824-d190a0005ebd-kube-api-access-8nvst\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.966957 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.966929 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:26.968914 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:26.968880 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:27.020056 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:27.020011 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvst\" (UniqueName: \"kubernetes.io/projected/52c794c5-eabb-4fad-b824-d190a0005ebd-kube-api-access-8nvst\") pod \"maas-api-revoke-sub-ae3d62bb-2bxhd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:27.304223 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:27.304172 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:27.676318 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:27.676249 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd"] Nov 02 21:44:27.676873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:27.676847 5800 scope.go:122] "RemoveContainer" containerID="e9e7be7cf18331a7054e61d8947623fee1dee3ae90e4fbe8fe1c188d31c66c4c" Nov 02 21:44:27.677825 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:44:27.677796 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c794c5_eabb_4fad_b824_d190a0005ebd.slice/crio-edbd98dd047b720d86bed244f7a96bcc413209186fcc98d4a8f777bbd3e4435f WatchSource:0}: Error finding container edbd98dd047b720d86bed244f7a96bcc413209186fcc98d4a8f777bbd3e4435f: Status 404 returned error can't find the container with id edbd98dd047b720d86bed244f7a96bcc413209186fcc98d4a8f777bbd3e4435f Nov 02 21:44:28.509368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:28.509315 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" event={"ID":"52c794c5-eabb-4fad-b824-d190a0005ebd","Type":"ContainerStarted","Data":"164a9ca35f62e6dd165cf7dfa4689f6a981deb611cc78a5a5983037866f280e9"} Nov 02 21:44:28.509821 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:28.509429 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" event={"ID":"52c794c5-eabb-4fad-b824-d190a0005ebd","Type":"ContainerStarted","Data":"edbd98dd047b720d86bed244f7a96bcc413209186fcc98d4a8f777bbd3e4435f"} Nov 02 21:44:28.617669 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:28.617609 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" podStartSLOduration=2.617592784 podStartE2EDuration="2.617592784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:28.614900121 +0000 UTC m=+6967.131672707" watchObservedRunningTime="2026-11-02 21:44:28.617592784 +0000 UTC m=+6967.134365370" Nov 02 21:44:28.698138 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:44:28.698095 5800 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c794c5_eabb_4fad_b824_d190a0005ebd.slice/crio-164a9ca35f62e6dd165cf7dfa4689f6a981deb611cc78a5a5983037866f280e9.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:44:29.517258 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:29.517218 5800 generic.go:356] "Generic (PLEG): container finished" podID="52c794c5-eabb-4fad-b824-d190a0005ebd" containerID="164a9ca35f62e6dd165cf7dfa4689f6a981deb611cc78a5a5983037866f280e9" exitCode=0 Nov 02 21:44:29.517258 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:29.517260 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" event={"ID":"52c794c5-eabb-4fad-b824-d190a0005ebd","Type":"ContainerDied","Data":"164a9ca35f62e6dd165cf7dfa4689f6a981deb611cc78a5a5983037866f280e9"} Nov 02 21:44:30.694696 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.694663 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:30.807147 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.807055 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-xks-ca\") pod \"52c794c5-eabb-4fad-b824-d190a0005ebd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " Nov 02 21:44:30.807147 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.807141 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c794c5-eabb-4fad-b824-d190a0005ebd-kube-api-access-8nvst\" (UniqueName: \"kubernetes.io/projected/52c794c5-eabb-4fad-b824-d190a0005ebd-kube-api-access-8nvst\") pod \"52c794c5-eabb-4fad-b824-d190a0005ebd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " Nov 02 21:44:30.807457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.807228 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-openshift-ca\") pod \"52c794c5-eabb-4fad-b824-d190a0005ebd\" (UID: \"52c794c5-eabb-4fad-b824-d190a0005ebd\") " Nov 02 21:44:30.807687 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.807655 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-openshift-ca" pod "52c794c5-eabb-4fad-b824-d190a0005ebd" (UID: "52c794c5-eabb-4fad-b824-d190a0005ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:44:30.810059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.810031 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-xks-ca" pod "52c794c5-eabb-4fad-b824-d190a0005ebd" (UID: "52c794c5-eabb-4fad-b824-d190a0005ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:30.810191 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.810031 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c794c5-eabb-4fad-b824-d190a0005ebd-kube-api-access-8nvst" pod "52c794c5-eabb-4fad-b824-d190a0005ebd" (UID: "52c794c5-eabb-4fad-b824-d190a0005ebd"). InnerVolumeSpecName "kube-api-access-8nvst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:30.908873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.908821 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:30.908873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.908865 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nvst\" (UniqueName: \"kubernetes.io/projected/52c794c5-eabb-4fad-b824-d190a0005ebd-kube-api-access-8nvst\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:30.908873 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:30.908876 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c794c5-eabb-4fad-b824-d190a0005ebd-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:31.532296 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:31.532256 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" Nov 02 21:44:31.532537 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:31.532315 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3d62bb-2bxhd" event={"ID":"52c794c5-eabb-4fad-b824-d190a0005ebd","Type":"ContainerDied","Data":"edbd98dd047b720d86bed244f7a96bcc413209186fcc98d4a8f777bbd3e4435f"} Nov 02 21:44:31.532537 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:31.532372 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edbd98dd047b720d86bed244f7a96bcc413209186fcc98d4a8f777bbd3e4435f" Nov 02 21:44:37.805936 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.805887 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf"] Nov 02 21:44:37.806571 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.806551 5800 cpu_manager.go:414] "containerMap: removing container" podUID="52c794c5-eabb-4fad-b824-d190a0005ebd" containerName="revoke-keys" Nov 02 21:44:37.806637 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.806574 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c794c5-eabb-4fad-b824-d190a0005ebd" containerName="revoke-keys" Nov 02 21:44:37.806729 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.806717 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c794c5-eabb-4fad-b824-d190a0005ebd" containerName="revoke-keys" Nov 02 21:44:37.810639 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.810610 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.860466 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.860424 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf"] Nov 02 21:44:37.891135 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.891075 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.891348 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.891270 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkmf\" (UniqueName: \"kubernetes.io/projected/b0a8918e-1b0c-4df3-85e0-58eab517644e-kube-api-access-xqkmf\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.891468 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.891397 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.902867 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.902828 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:44:37.903065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.902916 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:44:37.903065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.902980 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:44:37.905523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.905500 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:44:37.992439 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.992392 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkmf\" (UniqueName: \"kubernetes.io/projected/b0a8918e-1b0c-4df3-85e0-58eab517644e-kube-api-access-xqkmf\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.992694 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.992494 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.992694 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.992553 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.993368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.993342 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:37.995175 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:37.995150 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:38.046162 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:38.046125 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkmf\" (UniqueName: \"kubernetes.io/projected/b0a8918e-1b0c-4df3-85e0-58eab517644e-kube-api-access-xqkmf\") pod \"maas-api-revoke-sub-8371c064-4vdgf\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:38.127974 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:38.127878 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:38.499761 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:38.499716 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf"] Nov 02 21:44:38.501468 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:44:38.501437 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a8918e_1b0c_4df3_85e0_58eab517644e.slice/crio-f6598c0aa78b6827b21dbd62fa568a17eed17176eb47259a47163b19ceca4bbc WatchSource:0}: Error finding container f6598c0aa78b6827b21dbd62fa568a17eed17176eb47259a47163b19ceca4bbc: Status 404 returned error can't find the container with id f6598c0aa78b6827b21dbd62fa568a17eed17176eb47259a47163b19ceca4bbc Nov 02 21:44:38.575433 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:38.575392 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" event={"ID":"b0a8918e-1b0c-4df3-85e0-58eab517644e","Type":"ContainerStarted","Data":"f6598c0aa78b6827b21dbd62fa568a17eed17176eb47259a47163b19ceca4bbc"} Nov 02 21:44:39.583427 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:39.583380 5800 generic.go:356] "Generic (PLEG): container finished" podID="b0a8918e-1b0c-4df3-85e0-58eab517644e" containerID="b4007a7ae5c1df22218a8bd36c448755e9b942af70efc5195e10d8d0b2de58f7" exitCode=0 Nov 02 21:44:39.583974 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:39.583467 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" event={"ID":"b0a8918e-1b0c-4df3-85e0-58eab517644e","Type":"ContainerDied","Data":"b4007a7ae5c1df22218a8bd36c448755e9b942af70efc5195e10d8d0b2de58f7"} Nov 02 21:44:40.756630 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.756595 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:40.821686 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.821646 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-openshift-ca\") pod \"b0a8918e-1b0c-4df3-85e0-58eab517644e\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " Nov 02 21:44:40.821875 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.821729 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-xks-ca\") pod \"b0a8918e-1b0c-4df3-85e0-58eab517644e\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " Nov 02 21:44:40.821875 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.821758 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a8918e-1b0c-4df3-85e0-58eab517644e-kube-api-access-xqkmf\" (UniqueName: \"kubernetes.io/projected/b0a8918e-1b0c-4df3-85e0-58eab517644e-kube-api-access-xqkmf\") pod \"b0a8918e-1b0c-4df3-85e0-58eab517644e\" (UID: \"b0a8918e-1b0c-4df3-85e0-58eab517644e\") " Nov 02 21:44:40.822147 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.822119 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-openshift-ca" pod "b0a8918e-1b0c-4df3-85e0-58eab517644e" (UID: "b0a8918e-1b0c-4df3-85e0-58eab517644e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:44:40.824031 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.824003 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-xks-ca" pod "b0a8918e-1b0c-4df3-85e0-58eab517644e" (UID: "b0a8918e-1b0c-4df3-85e0-58eab517644e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:40.824111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.824082 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a8918e-1b0c-4df3-85e0-58eab517644e-kube-api-access-xqkmf" pod "b0a8918e-1b0c-4df3-85e0-58eab517644e" (UID: "b0a8918e-1b0c-4df3-85e0-58eab517644e"). InnerVolumeSpecName "kube-api-access-xqkmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:40.922805 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.922705 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:40.922805 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.922745 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqkmf\" (UniqueName: \"kubernetes.io/projected/b0a8918e-1b0c-4df3-85e0-58eab517644e-kube-api-access-xqkmf\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:40.922805 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:40.922756 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a8918e-1b0c-4df3-85e0-58eab517644e-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:41.628321 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:41.628256 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" Nov 02 21:44:41.631476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:41.631438 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8371c064-4vdgf" event={"ID":"b0a8918e-1b0c-4df3-85e0-58eab517644e","Type":"ContainerDied","Data":"f6598c0aa78b6827b21dbd62fa568a17eed17176eb47259a47163b19ceca4bbc"} Nov 02 21:44:41.631671 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:41.631501 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6598c0aa78b6827b21dbd62fa568a17eed17176eb47259a47163b19ceca4bbc" Nov 02 21:44:48.466295 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.466249 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2"] Nov 02 21:44:48.466698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.466674 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b0a8918e-1b0c-4df3-85e0-58eab517644e" containerName="revoke-keys" Nov 02 21:44:48.466698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.466686 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a8918e-1b0c-4df3-85e0-58eab517644e" containerName="revoke-keys" Nov 02 21:44:48.466799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.466775 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a8918e-1b0c-4df3-85e0-58eab517644e" containerName="revoke-keys" Nov 02 21:44:48.470215 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.470185 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.519647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.519586 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:44:48.519647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.519586 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:44:48.519916 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.519596 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:44:48.519983 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.519924 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:44:48.523650 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.523622 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2"] Nov 02 21:44:48.645549 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.645453 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jxv\" (UniqueName: \"kubernetes.io/projected/2d90241c-7f24-4753-8f27-afb25eb651d0-kube-api-access-x2jxv\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.645760 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.645655 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.645760 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.645698 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.746720 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.746614 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.746720 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.746667 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.746720 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.746704 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jxv\" (UniqueName: \"kubernetes.io/projected/2d90241c-7f24-4753-8f27-afb25eb651d0-kube-api-access-x2jxv\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.747458 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.747432 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.749236 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.749200 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:48.800058 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:48.800018 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jxv\" (UniqueName: \"kubernetes.io/projected/2d90241c-7f24-4753-8f27-afb25eb651d0-kube-api-access-x2jxv\") pod \"maas-api-revoke-sub-c63832cf-w2vk2\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:49.083561 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:49.083525 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:49.451375 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:49.451346 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2"] Nov 02 21:44:49.452580 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:44:49.452547 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d90241c_7f24_4753_8f27_afb25eb651d0.slice/crio-35616fbba5e4b0d3be06436810b2ba6cda2966c036bbd134e6cfce9fcd3c4ddb WatchSource:0}: Error finding container 35616fbba5e4b0d3be06436810b2ba6cda2966c036bbd134e6cfce9fcd3c4ddb: Status 404 returned error can't find the container with id 35616fbba5e4b0d3be06436810b2ba6cda2966c036bbd134e6cfce9fcd3c4ddb Nov 02 21:44:49.714858 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:49.714769 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" event={"ID":"2d90241c-7f24-4753-8f27-afb25eb651d0","Type":"ContainerStarted","Data":"35616fbba5e4b0d3be06436810b2ba6cda2966c036bbd134e6cfce9fcd3c4ddb"} Nov 02 21:44:50.722026 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:50.721930 5800 generic.go:356] "Generic (PLEG): container finished" podID="2d90241c-7f24-4753-8f27-afb25eb651d0" containerID="f68cbeb110e41ab56b68430999651fa639284166ecf53e62aee9fcd7b479afc7" exitCode=0 Nov 02 21:44:50.722026 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:50.722011 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" event={"ID":"2d90241c-7f24-4753-8f27-afb25eb651d0","Type":"ContainerDied","Data":"f68cbeb110e41ab56b68430999651fa639284166ecf53e62aee9fcd7b479afc7"} Nov 02 21:44:51.870706 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:51.870677 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:51.979985 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:51.979884 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-openshift-ca\") pod \"2d90241c-7f24-4753-8f27-afb25eb651d0\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " Nov 02 21:44:51.980164 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:51.980029 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d90241c-7f24-4753-8f27-afb25eb651d0-kube-api-access-x2jxv\" (UniqueName: \"kubernetes.io/projected/2d90241c-7f24-4753-8f27-afb25eb651d0-kube-api-access-x2jxv\") pod \"2d90241c-7f24-4753-8f27-afb25eb651d0\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " Nov 02 21:44:51.980164 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:51.980074 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-xks-ca\") pod \"2d90241c-7f24-4753-8f27-afb25eb651d0\" (UID: \"2d90241c-7f24-4753-8f27-afb25eb651d0\") " Nov 02 21:44:51.980521 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:51.980492 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-openshift-ca" pod "2d90241c-7f24-4753-8f27-afb25eb651d0" (UID: "2d90241c-7f24-4753-8f27-afb25eb651d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:44:51.982487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:51.982453 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-xks-ca" pod "2d90241c-7f24-4753-8f27-afb25eb651d0" (UID: "2d90241c-7f24-4753-8f27-afb25eb651d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:44:51.982614 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:51.982510 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d90241c-7f24-4753-8f27-afb25eb651d0-kube-api-access-x2jxv" pod "2d90241c-7f24-4753-8f27-afb25eb651d0" (UID: "2d90241c-7f24-4753-8f27-afb25eb651d0"). InnerVolumeSpecName "kube-api-access-x2jxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:44:52.081388 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:52.081353 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jxv\" (UniqueName: \"kubernetes.io/projected/2d90241c-7f24-4753-8f27-afb25eb651d0-kube-api-access-x2jxv\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:52.081388 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:52.081382 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:52.081388 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:52.081393 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d90241c-7f24-4753-8f27-afb25eb651d0-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:44:52.697514 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:44:52.697473 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:44:52.734710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:52.734674 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" Nov 02 21:44:52.734710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:52.734695 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63832cf-w2vk2" event={"ID":"2d90241c-7f24-4753-8f27-afb25eb651d0","Type":"ContainerDied","Data":"35616fbba5e4b0d3be06436810b2ba6cda2966c036bbd134e6cfce9fcd3c4ddb"} Nov 02 21:44:52.734934 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:52.734747 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35616fbba5e4b0d3be06436810b2ba6cda2966c036bbd134e6cfce9fcd3c4ddb" Nov 02 21:44:59.328559 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.328518 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb"] Nov 02 21:44:59.329078 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.328977 5800 cpu_manager.go:414] "containerMap: removing container" podUID="2d90241c-7f24-4753-8f27-afb25eb651d0" containerName="revoke-keys" Nov 02 21:44:59.329078 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.328990 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d90241c-7f24-4753-8f27-afb25eb651d0" containerName="revoke-keys" Nov 02 21:44:59.329173 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.329095 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d90241c-7f24-4753-8f27-afb25eb651d0" containerName="revoke-keys" Nov 02 21:44:59.332698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.332670 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.348920 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.348895 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.349054 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.348947 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgr78\" (UniqueName: \"kubernetes.io/projected/abb809b2-9743-4dd3-9dc3-683aae286822-kube-api-access-bgr78\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.349131 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.349106 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.384064 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.384026 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb"] Nov 02 21:44:59.426502 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.426448 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:44:59.426502 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.426481 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:44:59.428314 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.428290 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:44:59.428478 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.428325 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:44:59.450306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.450261 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.450460 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.450334 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgr78\" (UniqueName: \"kubernetes.io/projected/abb809b2-9743-4dd3-9dc3-683aae286822-kube-api-access-bgr78\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.450460 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.450386 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.450883 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.450861 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.452803 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.452776 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.503432 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.503392 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgr78\" (UniqueName: \"kubernetes.io/projected/abb809b2-9743-4dd3-9dc3-683aae286822-kube-api-access-bgr78\") pod \"maas-api-revoke-sub-e88e0e13-kxptb\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:44:59.646523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:44:59.646423 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:45:00.018339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:00.018311 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb"] Nov 02 21:45:00.019409 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:45:00.019374 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb809b2_9743_4dd3_9dc3_683aae286822.slice/crio-52687817f6c9665cc4d26fa8fff97074460631b47465a0335c23cfe22b02075c WatchSource:0}: Error finding container 52687817f6c9665cc4d26fa8fff97074460631b47465a0335c23cfe22b02075c: Status 404 returned error can't find the container with id 52687817f6c9665cc4d26fa8fff97074460631b47465a0335c23cfe22b02075c Nov 02 21:45:00.064465 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:00.064426 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp"] Nov 02 21:45:00.066863 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:00.066839 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-799zp"] Nov 02 21:45:00.788326 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:00.788273 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" event={"ID":"abb809b2-9743-4dd3-9dc3-683aae286822","Type":"ContainerStarted","Data":"07c3cdde053525f933df87a1f4b72805b18fd0715d441c26d1de7e422933cce1"} Nov 02 21:45:00.788759 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:00.788338 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" event={"ID":"abb809b2-9743-4dd3-9dc3-683aae286822","Type":"ContainerStarted","Data":"52687817f6c9665cc4d26fa8fff97074460631b47465a0335c23cfe22b02075c"} Nov 02 21:45:00.895638 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:00.895589 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" podStartSLOduration=1.895574119 podStartE2EDuration="1.895574119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:00.89271828 +0000 UTC m=+6999.409490864" watchObservedRunningTime="2026-11-02 21:45:00.895574119 +0000 UTC m=+6999.412346713" Nov 02 21:45:01.621607 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:01.621574 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b" path="/var/lib/kubelet/pods/e0bd55cc-1a37-4a9e-94fb-d25db6d6e15b/volumes" Nov 02 21:45:01.797395 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:01.797350 5800 generic.go:356] "Generic (PLEG): container finished" podID="abb809b2-9743-4dd3-9dc3-683aae286822" containerID="07c3cdde053525f933df87a1f4b72805b18fd0715d441c26d1de7e422933cce1" exitCode=0 Nov 02 21:45:01.797935 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:01.797447 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" event={"ID":"abb809b2-9743-4dd3-9dc3-683aae286822","Type":"ContainerDied","Data":"07c3cdde053525f933df87a1f4b72805b18fd0715d441c26d1de7e422933cce1"} Nov 02 21:45:03.205065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.204727 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:45:03.304315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.297959 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb809b2-9743-4dd3-9dc3-683aae286822-kube-api-access-bgr78\" (UniqueName: \"kubernetes.io/projected/abb809b2-9743-4dd3-9dc3-683aae286822-kube-api-access-bgr78\") pod \"abb809b2-9743-4dd3-9dc3-683aae286822\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " Nov 02 21:45:03.304315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.298019 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-xks-ca\") pod \"abb809b2-9743-4dd3-9dc3-683aae286822\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " Nov 02 21:45:03.304315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.298057 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-openshift-ca\") pod \"abb809b2-9743-4dd3-9dc3-683aae286822\" (UID: \"abb809b2-9743-4dd3-9dc3-683aae286822\") " Nov 02 21:45:03.304315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.298819 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-openshift-ca" pod "abb809b2-9743-4dd3-9dc3-683aae286822" (UID: "abb809b2-9743-4dd3-9dc3-683aae286822"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:03.305315 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.304774 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb809b2-9743-4dd3-9dc3-683aae286822-kube-api-access-bgr78" pod "abb809b2-9743-4dd3-9dc3-683aae286822" (UID: "abb809b2-9743-4dd3-9dc3-683aae286822"). InnerVolumeSpecName "kube-api-access-bgr78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:03.305993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.305969 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-xks-ca" pod "abb809b2-9743-4dd3-9dc3-683aae286822" (UID: "abb809b2-9743-4dd3-9dc3-683aae286822"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:03.400043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.399899 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgr78\" (UniqueName: \"kubernetes.io/projected/abb809b2-9743-4dd3-9dc3-683aae286822-kube-api-access-bgr78\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:03.400043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.399944 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:03.400043 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.399961 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abb809b2-9743-4dd3-9dc3-683aae286822-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:03.820306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.819467 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" event={"ID":"abb809b2-9743-4dd3-9dc3-683aae286822","Type":"ContainerDied","Data":"52687817f6c9665cc4d26fa8fff97074460631b47465a0335c23cfe22b02075c"} Nov 02 21:45:03.820306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.819542 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52687817f6c9665cc4d26fa8fff97074460631b47465a0335c23cfe22b02075c" Nov 02 21:45:03.820306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:03.819628 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e88e0e13-kxptb" Nov 02 21:45:10.048214 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.048176 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq"] Nov 02 21:45:10.048665 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.048650 5800 cpu_manager.go:414] "containerMap: removing container" podUID="abb809b2-9743-4dd3-9dc3-683aae286822" containerName="revoke-keys" Nov 02 21:45:10.048713 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.048667 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb809b2-9743-4dd3-9dc3-683aae286822" containerName="revoke-keys" Nov 02 21:45:10.048784 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.048774 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb809b2-9743-4dd3-9dc3-683aae286822" containerName="revoke-keys" Nov 02 21:45:10.052097 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.052080 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.075688 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.075658 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.075857 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.075710 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt4jp\" (UniqueName: \"kubernetes.io/projected/48af24b4-860c-4330-8acd-10ac9f48637f-kube-api-access-dt4jp\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.075857 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.075822 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.102697 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.102663 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:10.148647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.148602 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:45:10.148847 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.148696 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:10.148847 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.148741 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:45:10.151577 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.151554 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq"] Nov 02 21:45:10.176846 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.176810 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4jp\" (UniqueName: \"kubernetes.io/projected/48af24b4-860c-4330-8acd-10ac9f48637f-kube-api-access-dt4jp\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.177028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.176869 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.177028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.176940 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.177659 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.177631 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.179275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.179255 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.230689 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.230658 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4jp\" (UniqueName: \"kubernetes.io/projected/48af24b4-860c-4330-8acd-10ac9f48637f-kube-api-access-dt4jp\") pod \"maas-api-revoke-sub-cf4d404e-zl2tq\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.365022 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.364931 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:10.732485 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.732456 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq"] Nov 02 21:45:10.734218 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:45:10.734191 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48af24b4_860c_4330_8acd_10ac9f48637f.slice/crio-5be7145c9e7a9e61683ea0e051bfeb37c474bd40ee9c20aba10dc7e32d3a76e6 WatchSource:0}: Error finding container 5be7145c9e7a9e61683ea0e051bfeb37c474bd40ee9c20aba10dc7e32d3a76e6: Status 404 returned error can't find the container with id 5be7145c9e7a9e61683ea0e051bfeb37c474bd40ee9c20aba10dc7e32d3a76e6 Nov 02 21:45:10.861237 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:10.861199 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" event={"ID":"48af24b4-860c-4330-8acd-10ac9f48637f","Type":"ContainerStarted","Data":"5be7145c9e7a9e61683ea0e051bfeb37c474bd40ee9c20aba10dc7e32d3a76e6"} Nov 02 21:45:11.870222 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:11.870140 5800 generic.go:356] "Generic (PLEG): container finished" podID="48af24b4-860c-4330-8acd-10ac9f48637f" containerID="3ed59c5701743c971f0893c92c699fc89a0093637f7edf9e6147419423618169" exitCode=0 Nov 02 21:45:11.870222 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:11.870176 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" event={"ID":"48af24b4-860c-4330-8acd-10ac9f48637f","Type":"ContainerDied","Data":"3ed59c5701743c971f0893c92c699fc89a0093637f7edf9e6147419423618169"} Nov 02 21:45:13.013232 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.013200 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:13.107048 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.107013 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-openshift-ca\") pod \"48af24b4-860c-4330-8acd-10ac9f48637f\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " Nov 02 21:45:13.107235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.107074 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48af24b4-860c-4330-8acd-10ac9f48637f-kube-api-access-dt4jp\" (UniqueName: \"kubernetes.io/projected/48af24b4-860c-4330-8acd-10ac9f48637f-kube-api-access-dt4jp\") pod \"48af24b4-860c-4330-8acd-10ac9f48637f\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " Nov 02 21:45:13.107235 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.107111 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-xks-ca\") pod \"48af24b4-860c-4330-8acd-10ac9f48637f\" (UID: \"48af24b4-860c-4330-8acd-10ac9f48637f\") " Nov 02 21:45:13.107513 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.107478 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-openshift-ca" pod "48af24b4-860c-4330-8acd-10ac9f48637f" (UID: "48af24b4-860c-4330-8acd-10ac9f48637f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:13.109275 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.109246 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-xks-ca" pod "48af24b4-860c-4330-8acd-10ac9f48637f" (UID: "48af24b4-860c-4330-8acd-10ac9f48637f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:13.109425 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.109381 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48af24b4-860c-4330-8acd-10ac9f48637f-kube-api-access-dt4jp" pod "48af24b4-860c-4330-8acd-10ac9f48637f" (UID: "48af24b4-860c-4330-8acd-10ac9f48637f"). InnerVolumeSpecName "kube-api-access-dt4jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:13.208382 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.208299 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:13.208382 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.208331 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af24b4-860c-4330-8acd-10ac9f48637f-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:13.208382 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.208342 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt4jp\" (UniqueName: \"kubernetes.io/projected/48af24b4-860c-4330-8acd-10ac9f48637f-kube-api-access-dt4jp\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:13.884398 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.884313 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" Nov 02 21:45:13.884553 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.884317 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4d404e-zl2tq" event={"ID":"48af24b4-860c-4330-8acd-10ac9f48637f","Type":"ContainerDied","Data":"5be7145c9e7a9e61683ea0e051bfeb37c474bd40ee9c20aba10dc7e32d3a76e6"} Nov 02 21:45:13.884553 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:13.884442 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be7145c9e7a9e61683ea0e051bfeb37c474bd40ee9c20aba10dc7e32d3a76e6" Nov 02 21:45:20.044229 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.044188 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292"] Nov 02 21:45:20.090553 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.090518 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2m292"] Nov 02 21:45:20.719935 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.719899 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz"] Nov 02 21:45:20.720339 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.720326 5800 cpu_manager.go:414] "containerMap: removing container" podUID="48af24b4-860c-4330-8acd-10ac9f48637f" containerName="revoke-keys" Nov 02 21:45:20.720396 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.720340 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48af24b4-860c-4330-8acd-10ac9f48637f" containerName="revoke-keys" Nov 02 21:45:20.720449 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.720439 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="48af24b4-860c-4330-8acd-10ac9f48637f" containerName="revoke-keys" Nov 02 21:45:20.723736 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.723718 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.750918 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.750887 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46895aa-0775-42eb-acce-655661d8e349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.751045 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.750943 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l85d\" (UniqueName: \"kubernetes.io/projected/b46895aa-0775-42eb-acce-655661d8e349-kube-api-access-8l85d\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.751110 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.751088 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46895aa-0775-42eb-acce-655661d8e349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.773130 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.773095 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:45:20.776535 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.776507 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz"] Nov 02 21:45:20.819402 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.819366 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:20.819618 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.819444 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:20.819618 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.819524 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:45:20.852076 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.852037 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46895aa-0775-42eb-acce-655661d8e349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.852247 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.852094 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46895aa-0775-42eb-acce-655661d8e349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.852247 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.852129 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l85d\" (UniqueName: \"kubernetes.io/projected/b46895aa-0775-42eb-acce-655661d8e349-kube-api-access-8l85d\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.852805 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.852782 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46895aa-0775-42eb-acce-655661d8e349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.854516 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.854494 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46895aa-0775-42eb-acce-655661d8e349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:20.905487 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:20.905455 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l85d\" (UniqueName: \"kubernetes.io/projected/b46895aa-0775-42eb-acce-655661d8e349-kube-api-access-8l85d\") pod \"maas-api-revoke-sub-91b22158-qsbnz\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:21.037267 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:21.037232 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:21.402338 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:21.402306 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz"] Nov 02 21:45:21.404246 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:45:21.404215 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46895aa_0775_42eb_acce_655661d8e349.slice/crio-7fdec77c727c9aebb5e9c7e696edb7f8e72f645ed6a5c98e34d4edd20e2dec00 WatchSource:0}: Error finding container 7fdec77c727c9aebb5e9c7e696edb7f8e72f645ed6a5c98e34d4edd20e2dec00: Status 404 returned error can't find the container with id 7fdec77c727c9aebb5e9c7e696edb7f8e72f645ed6a5c98e34d4edd20e2dec00 Nov 02 21:45:21.620900 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:21.620867 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663fb314-c00e-4edf-806f-e4c7ccc7cd7e" path="/var/lib/kubelet/pods/663fb314-c00e-4edf-806f-e4c7ccc7cd7e/volumes" Nov 02 21:45:21.938599 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:21.938567 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" event={"ID":"b46895aa-0775-42eb-acce-655661d8e349","Type":"ContainerStarted","Data":"cdc5d67c895678dc4ecbebbe79964d8903b24b9e5e8e2f2841eef47bde783c09"} Nov 02 21:45:21.938785 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:21.938621 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" event={"ID":"b46895aa-0775-42eb-acce-655661d8e349","Type":"ContainerStarted","Data":"7fdec77c727c9aebb5e9c7e696edb7f8e72f645ed6a5c98e34d4edd20e2dec00"} Nov 02 21:45:22.044922 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:22.044868 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" podStartSLOduration=2.044853722 podStartE2EDuration="2.044853722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:22.042810972 +0000 UTC m=+7020.559583557" watchObservedRunningTime="2026-11-02 21:45:22.044853722 +0000 UTC m=+7020.561626306" Nov 02 21:45:22.696156 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:45:22.696105 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:45:22.946319 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:22.946215 5800 generic.go:356] "Generic (PLEG): container finished" podID="b46895aa-0775-42eb-acce-655661d8e349" containerID="cdc5d67c895678dc4ecbebbe79964d8903b24b9e5e8e2f2841eef47bde783c09" exitCode=0 Nov 02 21:45:22.946482 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:22.946317 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" event={"ID":"b46895aa-0775-42eb-acce-655661d8e349","Type":"ContainerDied","Data":"cdc5d67c895678dc4ecbebbe79964d8903b24b9e5e8e2f2841eef47bde783c09"} Nov 02 21:45:24.095544 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.095518 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:24.185227 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.185191 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46895aa-0775-42eb-acce-655661d8e349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46895aa-0775-42eb-acce-655661d8e349-maas-api-xks-ca\") pod \"b46895aa-0775-42eb-acce-655661d8e349\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " Nov 02 21:45:24.185446 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.185369 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46895aa-0775-42eb-acce-655661d8e349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46895aa-0775-42eb-acce-655661d8e349-maas-api-openshift-ca\") pod \"b46895aa-0775-42eb-acce-655661d8e349\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " Nov 02 21:45:24.185446 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.185396 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46895aa-0775-42eb-acce-655661d8e349-kube-api-access-8l85d\" (UniqueName: \"kubernetes.io/projected/b46895aa-0775-42eb-acce-655661d8e349-kube-api-access-8l85d\") pod \"b46895aa-0775-42eb-acce-655661d8e349\" (UID: \"b46895aa-0775-42eb-acce-655661d8e349\") " Nov 02 21:45:24.185775 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.185743 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46895aa-0775-42eb-acce-655661d8e349-maas-api-openshift-ca" pod "b46895aa-0775-42eb-acce-655661d8e349" (UID: "b46895aa-0775-42eb-acce-655661d8e349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:24.187461 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.187433 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46895aa-0775-42eb-acce-655661d8e349-maas-api-xks-ca" pod "b46895aa-0775-42eb-acce-655661d8e349" (UID: "b46895aa-0775-42eb-acce-655661d8e349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:24.187557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.187497 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46895aa-0775-42eb-acce-655661d8e349-kube-api-access-8l85d" pod "b46895aa-0775-42eb-acce-655661d8e349" (UID: "b46895aa-0775-42eb-acce-655661d8e349"). InnerVolumeSpecName "kube-api-access-8l85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:24.286698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.286660 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l85d\" (UniqueName: \"kubernetes.io/projected/b46895aa-0775-42eb-acce-655661d8e349-kube-api-access-8l85d\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:24.286698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.286694 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46895aa-0775-42eb-acce-655661d8e349-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:24.286698 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.286705 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46895aa-0775-42eb-acce-655661d8e349-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:24.960307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.960258 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" event={"ID":"b46895aa-0775-42eb-acce-655661d8e349","Type":"ContainerDied","Data":"7fdec77c727c9aebb5e9c7e696edb7f8e72f645ed6a5c98e34d4edd20e2dec00"} Nov 02 21:45:24.960469 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.960329 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fdec77c727c9aebb5e9c7e696edb7f8e72f645ed6a5c98e34d4edd20e2dec00" Nov 02 21:45:24.960469 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:24.960266 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91b22158-qsbnz" Nov 02 21:45:31.437437 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.437347 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4"] Nov 02 21:45:31.438019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.437819 5800 cpu_manager.go:414] "containerMap: removing container" podUID="b46895aa-0775-42eb-acce-655661d8e349" containerName="revoke-keys" Nov 02 21:45:31.438019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.437836 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46895aa-0775-42eb-acce-655661d8e349" containerName="revoke-keys" Nov 02 21:45:31.438019 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.437937 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46895aa-0775-42eb-acce-655661d8e349" containerName="revoke-keys" Nov 02 21:45:31.441631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.441605 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.494804 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.494767 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4"] Nov 02 21:45:31.537012 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.536966 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:45:31.537232 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.536992 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:31.537232 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.536998 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:31.537232 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.537074 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:45:31.566640 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.566592 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.566861 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.566720 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.566861 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.566772 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngg8\" (UniqueName: \"kubernetes.io/projected/9a9dfa7e-6f1f-467a-a707-829b7f48f286-kube-api-access-mngg8\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.667641 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.667599 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.667842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.667706 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.667842 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.667766 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngg8\" (UniqueName: \"kubernetes.io/projected/9a9dfa7e-6f1f-467a-a707-829b7f48f286-kube-api-access-mngg8\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.668453 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.668427 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.670101 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.670076 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.720938 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.720865 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngg8\" (UniqueName: \"kubernetes.io/projected/9a9dfa7e-6f1f-467a-a707-829b7f48f286-kube-api-access-mngg8\") pod \"maas-api-revoke-sub-1cb6659c-nhts4\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:31.755818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:31.755781 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:32.124761 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:32.124735 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4"] Nov 02 21:45:32.126481 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:45:32.126451 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9dfa7e_6f1f_467a_a707_829b7f48f286.slice/crio-9f94ee84c3c00a52a49c112ebddcc9c6be7adefd4941d8624160a3f2f5f5929c WatchSource:0}: Error finding container 9f94ee84c3c00a52a49c112ebddcc9c6be7adefd4941d8624160a3f2f5f5929c: Status 404 returned error can't find the container with id 9f94ee84c3c00a52a49c112ebddcc9c6be7adefd4941d8624160a3f2f5f5929c Nov 02 21:45:33.008870 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:33.008828 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" event={"ID":"9a9dfa7e-6f1f-467a-a707-829b7f48f286","Type":"ContainerStarted","Data":"e23f3d99cf084f80b061399b2fca31edeb833d0f4ee2be21fcb6ff0b03b2c0e1"} Nov 02 21:45:33.009420 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:33.008882 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" event={"ID":"9a9dfa7e-6f1f-467a-a707-829b7f48f286","Type":"ContainerStarted","Data":"9f94ee84c3c00a52a49c112ebddcc9c6be7adefd4941d8624160a3f2f5f5929c"} Nov 02 21:45:33.117523 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:33.117465 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" podStartSLOduration=2.117450147 podStartE2EDuration="2.117450147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:33.114344779 +0000 UTC m=+7031.631117365" watchObservedRunningTime="2026-11-02 21:45:33.117450147 +0000 UTC m=+7031.634222733" Nov 02 21:45:34.032576 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:34.032528 5800 generic.go:356] "Generic (PLEG): container finished" podID="9a9dfa7e-6f1f-467a-a707-829b7f48f286" containerID="e23f3d99cf084f80b061399b2fca31edeb833d0f4ee2be21fcb6ff0b03b2c0e1" exitCode=0 Nov 02 21:45:34.033014 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:34.032625 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" event={"ID":"9a9dfa7e-6f1f-467a-a707-829b7f48f286","Type":"ContainerDied","Data":"e23f3d99cf084f80b061399b2fca31edeb833d0f4ee2be21fcb6ff0b03b2c0e1"} Nov 02 21:45:35.417025 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.416998 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:35.513714 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.513679 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-openshift-ca\") pod \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " Nov 02 21:45:35.513888 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.513754 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9dfa7e-6f1f-467a-a707-829b7f48f286-kube-api-access-mngg8\" (UniqueName: \"kubernetes.io/projected/9a9dfa7e-6f1f-467a-a707-829b7f48f286-kube-api-access-mngg8\") pod \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " Nov 02 21:45:35.513888 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.513781 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-xks-ca\") pod \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\" (UID: \"9a9dfa7e-6f1f-467a-a707-829b7f48f286\") " Nov 02 21:45:35.514125 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.514097 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-openshift-ca" pod "9a9dfa7e-6f1f-467a-a707-829b7f48f286" (UID: "9a9dfa7e-6f1f-467a-a707-829b7f48f286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:35.514304 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.514272 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:35.516186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.516158 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-xks-ca" pod "9a9dfa7e-6f1f-467a-a707-829b7f48f286" (UID: "9a9dfa7e-6f1f-467a-a707-829b7f48f286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:35.516186 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.516168 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9dfa7e-6f1f-467a-a707-829b7f48f286-kube-api-access-mngg8" pod "9a9dfa7e-6f1f-467a-a707-829b7f48f286" (UID: "9a9dfa7e-6f1f-467a-a707-829b7f48f286"). InnerVolumeSpecName "kube-api-access-mngg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:35.616111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.616077 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a9dfa7e-6f1f-467a-a707-829b7f48f286-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:35.616111 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:35.616109 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngg8\" (UniqueName: \"kubernetes.io/projected/9a9dfa7e-6f1f-467a-a707-829b7f48f286-kube-api-access-mngg8\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:36.077305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:36.076999 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" event={"ID":"9a9dfa7e-6f1f-467a-a707-829b7f48f286","Type":"ContainerDied","Data":"9f94ee84c3c00a52a49c112ebddcc9c6be7adefd4941d8624160a3f2f5f5929c"} Nov 02 21:45:36.077305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:36.077070 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f94ee84c3c00a52a49c112ebddcc9c6be7adefd4941d8624160a3f2f5f5929c" Nov 02 21:45:36.077305 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:36.077144 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cb6659c-nhts4" Nov 02 21:45:42.149024 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.148989 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462"] Nov 02 21:45:42.149468 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.149453 5800 cpu_manager.go:414] "containerMap: removing container" podUID="9a9dfa7e-6f1f-467a-a707-829b7f48f286" containerName="revoke-keys" Nov 02 21:45:42.149511 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.149469 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9dfa7e-6f1f-467a-a707-829b7f48f286" containerName="revoke-keys" Nov 02 21:45:42.149557 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.149548 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9dfa7e-6f1f-467a-a707-829b7f48f286" containerName="revoke-keys" Nov 02 21:45:42.152796 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.152771 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.178768 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.178737 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.178903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.178786 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.178903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.178807 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sct5f\" (UniqueName: \"kubernetes.io/projected/073c601b-e8eb-462b-a646-cada4f9261ed-kube-api-access-sct5f\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.206476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.206443 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462"] Nov 02 21:45:42.238941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.238908 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:45:42.247637 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.247609 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:42.247773 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.247650 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:45:42.247773 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.247651 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:42.280027 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.279992 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.280184 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.280049 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.280184 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.280072 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sct5f\" (UniqueName: \"kubernetes.io/projected/073c601b-e8eb-462b-a646-cada4f9261ed-kube-api-access-sct5f\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.280806 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.280781 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.282531 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.282507 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.333028 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.332996 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sct5f\" (UniqueName: \"kubernetes.io/projected/073c601b-e8eb-462b-a646-cada4f9261ed-kube-api-access-sct5f\") pod \"maas-api-revoke-sub-6e95a2be-xp462\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.471444 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.471359 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:42.837850 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:42.837820 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462"] Nov 02 21:45:42.838776 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:45:42.838752 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073c601b_e8eb_462b_a646_cada4f9261ed.slice/crio-7b9fa1edef1715d2bc1778c863f769076db9e4760d87f89a407469985ddcdb19 WatchSource:0}: Error finding container 7b9fa1edef1715d2bc1778c863f769076db9e4760d87f89a407469985ddcdb19: Status 404 returned error can't find the container with id 7b9fa1edef1715d2bc1778c863f769076db9e4760d87f89a407469985ddcdb19 Nov 02 21:45:43.123032 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:43.122944 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" event={"ID":"073c601b-e8eb-462b-a646-cada4f9261ed","Type":"ContainerStarted","Data":"7b9fa1edef1715d2bc1778c863f769076db9e4760d87f89a407469985ddcdb19"} Nov 02 21:45:44.130253 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:44.130162 5800 generic.go:356] "Generic (PLEG): container finished" podID="073c601b-e8eb-462b-a646-cada4f9261ed" containerID="cfd60465b2dcd0763d4ef0a59d73647cf13bab1f8daa9037411eeba4be5f2850" exitCode=0 Nov 02 21:45:44.130658 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:44.130252 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" event={"ID":"073c601b-e8eb-462b-a646-cada4f9261ed","Type":"ContainerDied","Data":"cfd60465b2dcd0763d4ef0a59d73647cf13bab1f8daa9037411eeba4be5f2850"} Nov 02 21:45:45.327818 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.327788 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:45.415213 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.415109 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073c601b-e8eb-462b-a646-cada4f9261ed-kube-api-access-sct5f\" (UniqueName: \"kubernetes.io/projected/073c601b-e8eb-462b-a646-cada4f9261ed-kube-api-access-sct5f\") pod \"073c601b-e8eb-462b-a646-cada4f9261ed\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " Nov 02 21:45:45.415421 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.415342 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-xks-ca\") pod \"073c601b-e8eb-462b-a646-cada4f9261ed\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " Nov 02 21:45:45.415500 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.415420 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-openshift-ca\") pod \"073c601b-e8eb-462b-a646-cada4f9261ed\" (UID: \"073c601b-e8eb-462b-a646-cada4f9261ed\") " Nov 02 21:45:45.415919 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.415863 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-openshift-ca" pod "073c601b-e8eb-462b-a646-cada4f9261ed" (UID: "073c601b-e8eb-462b-a646-cada4f9261ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:45.417681 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.417639 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073c601b-e8eb-462b-a646-cada4f9261ed-kube-api-access-sct5f" pod "073c601b-e8eb-462b-a646-cada4f9261ed" (UID: "073c601b-e8eb-462b-a646-cada4f9261ed"). InnerVolumeSpecName "kube-api-access-sct5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:45.418036 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.418009 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-xks-ca" pod "073c601b-e8eb-462b-a646-cada4f9261ed" (UID: "073c601b-e8eb-462b-a646-cada4f9261ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:45.517628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.517583 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:45.517628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.517623 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073c601b-e8eb-462b-a646-cada4f9261ed-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:45.517885 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:45.517657 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sct5f\" (UniqueName: \"kubernetes.io/projected/073c601b-e8eb-462b-a646-cada4f9261ed-kube-api-access-sct5f\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:46.163306 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:46.162456 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" event={"ID":"073c601b-e8eb-462b-a646-cada4f9261ed","Type":"ContainerDied","Data":"7b9fa1edef1715d2bc1778c863f769076db9e4760d87f89a407469985ddcdb19"} Nov 02 21:45:46.168324 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:46.167387 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e95a2be-xp462" Nov 02 21:45:46.181303 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:46.173570 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9fa1edef1715d2bc1778c863f769076db9e4760d87f89a407469985ddcdb19" Nov 02 21:45:52.695917 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:45:52.695877 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:45:52.922480 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:52.922439 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn"] Nov 02 21:45:52.923145 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:52.923123 5800 cpu_manager.go:414] "containerMap: removing container" podUID="073c601b-e8eb-462b-a646-cada4f9261ed" containerName="revoke-keys" Nov 02 21:45:52.923198 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:52.923149 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073c601b-e8eb-462b-a646-cada4f9261ed" containerName="revoke-keys" Nov 02 21:45:52.923329 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:52.923317 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="073c601b-e8eb-462b-a646-cada4f9261ed" containerName="revoke-keys" Nov 02 21:45:52.927057 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:52.927039 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:52.978846 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:52.978761 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn"] Nov 02 21:45:53.020059 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.020021 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:45:53.020262 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.020096 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:45:53.020262 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.020162 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:45:53.020262 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.020198 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:45:53.031368 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.031335 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.031530 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.031402 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.031598 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.031543 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wk4g\" (UniqueName: \"kubernetes.io/projected/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-kube-api-access-4wk4g\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.132115 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.132076 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.132335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.132174 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wk4g\" (UniqueName: \"kubernetes.io/projected/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-kube-api-access-4wk4g\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.132335 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.132199 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.132911 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.132886 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.134618 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.134591 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.185500 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.185464 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wk4g\" (UniqueName: \"kubernetes.io/projected/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-kube-api-access-4wk4g\") pod \"maas-api-revoke-sub-ab40b961-m6qzn\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.244218 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.244130 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:53.615871 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:45:53.615835 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a0203d_d15b_4941_ab03_d0ab17bbffb9.slice/crio-120305133fd69ba388e1ea3ce23952072d44a8b3a4d132e44497e5ca30d7a1e5 WatchSource:0}: Error finding container 120305133fd69ba388e1ea3ce23952072d44a8b3a4d132e44497e5ca30d7a1e5: Status 404 returned error can't find the container with id 120305133fd69ba388e1ea3ce23952072d44a8b3a4d132e44497e5ca30d7a1e5 Nov 02 21:45:53.622828 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:53.622802 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn"] Nov 02 21:45:54.232956 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:54.232863 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" event={"ID":"c8a0203d-d15b-4941-ab03-d0ab17bbffb9","Type":"ContainerStarted","Data":"07cbcc2d1931024eda4ab2fa0c3c0fff4c71682d29e82ad3740e0368355171d0"} Nov 02 21:45:54.232956 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:54.232918 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" event={"ID":"c8a0203d-d15b-4941-ab03-d0ab17bbffb9","Type":"ContainerStarted","Data":"120305133fd69ba388e1ea3ce23952072d44a8b3a4d132e44497e5ca30d7a1e5"} Nov 02 21:45:54.340872 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:54.340808 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" podStartSLOduration=2.340793298 podStartE2EDuration="2.340793298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:54.337436799 +0000 UTC m=+7052.854209590" watchObservedRunningTime="2026-11-02 21:45:54.340793298 +0000 UTC m=+7052.857565882" Nov 02 21:45:55.245814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:55.245773 5800 generic.go:356] "Generic (PLEG): container finished" podID="c8a0203d-d15b-4941-ab03-d0ab17bbffb9" containerID="07cbcc2d1931024eda4ab2fa0c3c0fff4c71682d29e82ad3740e0368355171d0" exitCode=0 Nov 02 21:45:55.246238 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:55.245844 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" event={"ID":"c8a0203d-d15b-4941-ab03-d0ab17bbffb9","Type":"ContainerDied","Data":"07cbcc2d1931024eda4ab2fa0c3c0fff4c71682d29e82ad3740e0368355171d0"} Nov 02 21:45:56.812458 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:56.811639 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:56.916574 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:56.916536 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-openshift-ca\") pod \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " Nov 02 21:45:56.916798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:56.916596 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-xks-ca\") pod \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " Nov 02 21:45:56.916798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:56.916625 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-kube-api-access-4wk4g\" (UniqueName: \"kubernetes.io/projected/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-kube-api-access-4wk4g\") pod \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\" (UID: \"c8a0203d-d15b-4941-ab03-d0ab17bbffb9\") " Nov 02 21:45:56.917005 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:56.916978 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-openshift-ca" pod "c8a0203d-d15b-4941-ab03-d0ab17bbffb9" (UID: "c8a0203d-d15b-4941-ab03-d0ab17bbffb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:56.919314 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:56.919268 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-xks-ca" pod "c8a0203d-d15b-4941-ab03-d0ab17bbffb9" (UID: "c8a0203d-d15b-4941-ab03-d0ab17bbffb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:45:56.919597 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:56.919568 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-kube-api-access-4wk4g" pod "c8a0203d-d15b-4941-ab03-d0ab17bbffb9" (UID: "c8a0203d-d15b-4941-ab03-d0ab17bbffb9"). InnerVolumeSpecName "kube-api-access-4wk4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:57.017860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:57.017822 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.017860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:57.017856 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.017860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:57.017867 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wk4g\" (UniqueName: \"kubernetes.io/projected/c8a0203d-d15b-4941-ab03-d0ab17bbffb9-kube-api-access-4wk4g\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:57.294936 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:57.294660 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" Nov 02 21:45:57.294936 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:57.294679 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab40b961-m6qzn" event={"ID":"c8a0203d-d15b-4941-ab03-d0ab17bbffb9","Type":"ContainerDied","Data":"120305133fd69ba388e1ea3ce23952072d44a8b3a4d132e44497e5ca30d7a1e5"} Nov 02 21:45:57.294936 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:45:57.294739 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="120305133fd69ba388e1ea3ce23952072d44a8b3a4d132e44497e5ca30d7a1e5" Nov 02 21:46:00.043353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:00.043313 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7"] Nov 02 21:46:00.089941 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:00.089904 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-738331b1-gdpj7"] Nov 02 21:46:01.620418 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:01.620387 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2" path="/var/lib/kubelet/pods/365129d5-6f90-4ff3-bc9e-a8d7f5efa5a2/volumes" Nov 02 21:46:03.647310 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.647264 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m"] Nov 02 21:46:03.647814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.647725 5800 cpu_manager.go:414] "containerMap: removing container" podUID="c8a0203d-d15b-4941-ab03-d0ab17bbffb9" containerName="revoke-keys" Nov 02 21:46:03.647814 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.647736 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a0203d-d15b-4941-ab03-d0ab17bbffb9" containerName="revoke-keys" Nov 02 21:46:03.647891 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.647819 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a0203d-d15b-4941-ab03-d0ab17bbffb9" containerName="revoke-keys" Nov 02 21:46:03.651380 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.651361 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.690586 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.690550 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.690782 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.690624 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5tm8\" (UniqueName: \"kubernetes.io/projected/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-kube-api-access-d5tm8\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.690782 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.690714 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.705595 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.705556 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m"] Nov 02 21:46:03.747353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.747316 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:46:03.747353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.747321 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:46:03.747675 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.747413 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:46:03.747675 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.747414 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:46:03.791479 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.791432 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.791682 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.791516 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tm8\" (UniqueName: \"kubernetes.io/projected/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-kube-api-access-d5tm8\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.791682 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.791541 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.792118 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.792089 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.793913 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.793892 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.845038 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.845001 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tm8\" (UniqueName: \"kubernetes.io/projected/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-kube-api-access-d5tm8\") pod \"maas-api-revoke-sub-518f509c-hzd5m\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:03.964964 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:03.964859 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:04.335353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:04.335319 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m"] Nov 02 21:46:04.336835 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:46:04.336802 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e2afdc_55e1_4bc9_b669_d71c93e27ddd.slice/crio-f74085a6b6764b51723d4c24bdf509d6a01b47163b25b67ebd8ec141409151c3 WatchSource:0}: Error finding container f74085a6b6764b51723d4c24bdf509d6a01b47163b25b67ebd8ec141409151c3: Status 404 returned error can't find the container with id f74085a6b6764b51723d4c24bdf509d6a01b47163b25b67ebd8ec141409151c3 Nov 02 21:46:05.347783 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:05.347745 5800 generic.go:356] "Generic (PLEG): container finished" podID="29e2afdc-55e1-4bc9-b669-d71c93e27ddd" containerID="b675622ebb9a5ae2b944497bed69803f4dcd9828200fb28a45461d882d478cb5" exitCode=0 Nov 02 21:46:05.348225 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:05.347838 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" event={"ID":"29e2afdc-55e1-4bc9-b669-d71c93e27ddd","Type":"ContainerDied","Data":"b675622ebb9a5ae2b944497bed69803f4dcd9828200fb28a45461d882d478cb5"} Nov 02 21:46:05.348225 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:05.347905 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" event={"ID":"29e2afdc-55e1-4bc9-b669-d71c93e27ddd","Type":"ContainerStarted","Data":"f74085a6b6764b51723d4c24bdf509d6a01b47163b25b67ebd8ec141409151c3"} Nov 02 21:46:06.529928 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.529903 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:06.623491 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.623397 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-openshift-ca\") pod \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " Nov 02 21:46:06.623679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.623496 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-kube-api-access-d5tm8\" (UniqueName: \"kubernetes.io/projected/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-kube-api-access-d5tm8\") pod \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " Nov 02 21:46:06.623679 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.623572 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-xks-ca\") pod \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\" (UID: \"29e2afdc-55e1-4bc9-b669-d71c93e27ddd\") " Nov 02 21:46:06.623823 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.623800 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-openshift-ca" pod "29e2afdc-55e1-4bc9-b669-d71c93e27ddd" (UID: "29e2afdc-55e1-4bc9-b669-d71c93e27ddd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:06.623968 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.623933 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:06.625834 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.625804 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-kube-api-access-d5tm8" pod "29e2afdc-55e1-4bc9-b669-d71c93e27ddd" (UID: "29e2afdc-55e1-4bc9-b669-d71c93e27ddd"). InnerVolumeSpecName "kube-api-access-d5tm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:06.630421 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.630384 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-xks-ca" pod "29e2afdc-55e1-4bc9-b669-d71c93e27ddd" (UID: "29e2afdc-55e1-4bc9-b669-d71c93e27ddd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:06.725266 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.725223 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5tm8\" (UniqueName: \"kubernetes.io/projected/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-kube-api-access-d5tm8\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:06.725457 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:06.725269 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e2afdc-55e1-4bc9-b669-d71c93e27ddd-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:07.401022 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:07.400977 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" event={"ID":"29e2afdc-55e1-4bc9-b669-d71c93e27ddd","Type":"ContainerDied","Data":"f74085a6b6764b51723d4c24bdf509d6a01b47163b25b67ebd8ec141409151c3"} Nov 02 21:46:07.401248 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:07.401048 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f74085a6b6764b51723d4c24bdf509d6a01b47163b25b67ebd8ec141409151c3" Nov 02 21:46:07.401248 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:07.401101 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518f509c-hzd5m" Nov 02 21:46:08.316498 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:08.316440 5800 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:46:08.316917 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:08.316522 5800 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-h2gr5" podUID="6eaf89b5-5d8f-4515-9c00-cef33ea31033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:46:10.044713 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:10.044661 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds"] Nov 02 21:46:10.090906 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:10.090869 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-989f229b-kvpds"] Nov 02 21:46:11.622710 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:11.622676 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab68440-1897-4f50-86fa-9d2b80002e3c" path="/var/lib/kubelet/pods/1ab68440-1897-4f50-86fa-9d2b80002e3c/volumes" Nov 02 21:46:14.367986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.367945 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9"] Nov 02 21:46:14.368425 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.368379 5800 cpu_manager.go:414] "containerMap: removing container" podUID="29e2afdc-55e1-4bc9-b669-d71c93e27ddd" containerName="revoke-keys" Nov 02 21:46:14.368425 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.368391 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e2afdc-55e1-4bc9-b669-d71c93e27ddd" containerName="revoke-keys" Nov 02 21:46:14.368520 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.368474 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e2afdc-55e1-4bc9-b669-d71c93e27ddd" containerName="revoke-keys" Nov 02 21:46:14.371895 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.371878 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.426845 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.426809 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9"] Nov 02 21:46:14.467602 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.467554 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 21:46:14.467816 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.467615 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:46:14.467816 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.467629 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v6w4h\"" Nov 02 21:46:14.467816 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.467642 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:46:14.548662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.548626 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt68j\" (UniqueName: \"kubernetes.io/projected/d731c67d-1af9-4e46-b451-7d4de48e5147-kube-api-access-wt68j\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.548864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.548765 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.548864 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.548856 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.649847 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.649751 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.649847 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.649824 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.650117 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.649863 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt68j\" (UniqueName: \"kubernetes.io/projected/d731c67d-1af9-4e46-b451-7d4de48e5147-kube-api-access-wt68j\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.650624 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.650597 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.652423 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.652399 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.704360 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.704324 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt68j\" (UniqueName: \"kubernetes.io/projected/d731c67d-1af9-4e46-b451-7d4de48e5147-kube-api-access-wt68j\") pod \"maas-api-revoke-sub-c5c3e7de-cc9g9\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:14.985065 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:14.984967 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:15.353008 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:15.352980 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9"] Nov 02 21:46:15.354032 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:46:15.354009 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd731c67d_1af9_4e46_b451_7d4de48e5147.slice/crio-71f5a50300a8bdd00a20a2a0c31d4b5ee084845d61a89042832b4d2a3df814b5 WatchSource:0}: Error finding container 71f5a50300a8bdd00a20a2a0c31d4b5ee084845d61a89042832b4d2a3df814b5: Status 404 returned error can't find the container with id 71f5a50300a8bdd00a20a2a0c31d4b5ee084845d61a89042832b4d2a3df814b5 Nov 02 21:46:15.460553 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:15.460514 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" event={"ID":"d731c67d-1af9-4e46-b451-7d4de48e5147","Type":"ContainerStarted","Data":"71f5a50300a8bdd00a20a2a0c31d4b5ee084845d61a89042832b4d2a3df814b5"} Nov 02 21:46:16.469107 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:16.469013 5800 generic.go:356] "Generic (PLEG): container finished" podID="d731c67d-1af9-4e46-b451-7d4de48e5147" containerID="b6bf9c52432d733dafab311c2024493e6bd851ab6f7dbdbe8f21c6dd96daf07e" exitCode=0 Nov 02 21:46:16.469614 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:16.469097 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" event={"ID":"d731c67d-1af9-4e46-b451-7d4de48e5147","Type":"ContainerDied","Data":"b6bf9c52432d733dafab311c2024493e6bd851ab6f7dbdbe8f21c6dd96daf07e"} Nov 02 21:46:17.996620 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:17.996592 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:18.174999 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.174903 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-openshift-ca\") pod \"d731c67d-1af9-4e46-b451-7d4de48e5147\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " Nov 02 21:46:18.175766 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.175744 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d731c67d-1af9-4e46-b451-7d4de48e5147-kube-api-access-wt68j\" (UniqueName: \"kubernetes.io/projected/d731c67d-1af9-4e46-b451-7d4de48e5147-kube-api-access-wt68j\") pod \"d731c67d-1af9-4e46-b451-7d4de48e5147\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " Nov 02 21:46:18.176341 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.176321 5800 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-xks-ca\") pod \"d731c67d-1af9-4e46-b451-7d4de48e5147\" (UID: \"d731c67d-1af9-4e46-b451-7d4de48e5147\") " Nov 02 21:46:18.177061 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.175625 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-openshift-ca" pod "d731c67d-1af9-4e46-b451-7d4de48e5147" (UID: "d731c67d-1af9-4e46-b451-7d4de48e5147"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:18.198161 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.198112 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d731c67d-1af9-4e46-b451-7d4de48e5147-kube-api-access-wt68j" pod "d731c67d-1af9-4e46-b451-7d4de48e5147" (UID: "d731c67d-1af9-4e46-b451-7d4de48e5147"). InnerVolumeSpecName "kube-api-access-wt68j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:18.198414 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.198380 5800 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-xks-ca" pod "d731c67d-1af9-4e46-b451-7d4de48e5147" (UID: "d731c67d-1af9-4e46-b451-7d4de48e5147"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:46:18.297422 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.297325 5800 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt68j\" (UniqueName: \"kubernetes.io/projected/d731c67d-1af9-4e46-b451-7d4de48e5147-kube-api-access-wt68j\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:18.297832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.297792 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-xks-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:18.298026 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.297991 5800 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d731c67d-1af9-4e46-b451-7d4de48e5147-maas-api-openshift-ca\") on node \"ip-10-0-134-13.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:18.507264 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.507231 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" event={"ID":"d731c67d-1af9-4e46-b451-7d4de48e5147","Type":"ContainerDied","Data":"71f5a50300a8bdd00a20a2a0c31d4b5ee084845d61a89042832b4d2a3df814b5"} Nov 02 21:46:18.507502 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.507489 5800 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f5a50300a8bdd00a20a2a0c31d4b5ee084845d61a89042832b4d2a3df814b5" Nov 02 21:46:18.507655 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:18.507645 5800 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c3e7de-cc9g9" Nov 02 21:46:22.695951 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:46:22.695913 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:46:27.741307 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:27.734743 5800 scope.go:122] "RemoveContainer" containerID="3e202d53cca08a781ed93a9b0d79d698705269e59c6aab45ec596da3d797ea0e" Nov 02 21:46:27.804334 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:27.804306 5800 scope.go:122] "RemoveContainer" containerID="9600f464ee56908b841c1482ba162e34da625d2338eceaaf2bdebbd2896cf5f6" Nov 02 21:46:43.052108 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:43.052073 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92"] Nov 02 21:46:43.098593 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:43.098559 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea03a972-6dk92"] Nov 02 21:46:43.620799 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:43.620768 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432fc8ae-575e-408c-a181-a0f39dceb325" path="/var/lib/kubelet/pods/432fc8ae-575e-408c-a181-a0f39dceb325/volumes" Nov 02 21:46:44.368112 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:44.368079 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f74f7c786-fr9rr_44b1f0d7-2146-4520-bfc1-f7c56ddd37be/authorino/0.log" Nov 02 21:46:52.697472 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:46:52.697429 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:46:59.653098 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:59.653064 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-10d549e0-dvxrv_4131915b-6e43-4501-a28f-d80587a53c93/revoke-keys/0.log" Nov 02 21:46:59.882860 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:46:59.882827 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1cb6659c-nhts4_9a9dfa7e-6f1f-467a-a707-829b7f48f286/revoke-keys/0.log" Nov 02 21:47:00.101072 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:00.101042 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-518f509c-hzd5m_29e2afdc-55e1-4bc9-b669-d71c93e27ddd/revoke-keys/0.log" Nov 02 21:47:00.316367 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:00.316338 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6e95a2be-xp462_073c601b-e8eb-462b-a646-cada4f9261ed/revoke-keys/0.log" Nov 02 21:47:00.537441 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:00.537411 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7678f6d5-fpp9r_551c87f3-ac6e-4aef-97a7-9651d08f6a9b/revoke-keys/0.log" Nov 02 21:47:00.746840 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:00.746805 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8371c064-4vdgf_b0a8918e-1b0c-4df3-85e0-58eab517644e/revoke-keys/0.log" Nov 02 21:47:00.963040 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:00.962965 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-91b22158-qsbnz_b46895aa-0775-42eb-acce-655661d8e349/revoke-keys/0.log" Nov 02 21:47:01.192735 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:01.192703 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab40b961-m6qzn_c8a0203d-d15b-4941-ab03-d0ab17bbffb9/revoke-keys/0.log" Nov 02 21:47:01.405195 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:01.405160 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ae3d62bb-2bxhd_52c794c5-eabb-4fad-b824-d190a0005ebd/revoke-keys/0.log" Nov 02 21:47:01.626488 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:01.626456 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bbcf6212-mc8nz_84882581-4345-416a-8076-2bc35fa56aaa/revoke-keys/0.log" Nov 02 21:47:01.848647 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:01.848615 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c2db92a0-f88tk_9b5a0964-e15b-4b4a-8163-5a60fae981b7/revoke-keys/0.log" Nov 02 21:47:02.052239 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:02.052213 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c5c3e7de-cc9g9_d731c67d-1af9-4e46-b451-7d4de48e5147/revoke-keys/0.log" Nov 02 21:47:02.273420 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:02.273377 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c63832cf-w2vk2_2d90241c-7f24-4753-8f27-afb25eb651d0/revoke-keys/0.log" Nov 02 21:47:02.499528 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:02.499495 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cf4d404e-zl2tq_48af24b4-860c-4330-8acd-10ac9f48637f/revoke-keys/0.log" Nov 02 21:47:02.715423 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:02.715342 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-db968f7a-gwltq_bd029bea-c23a-440f-a179-f08627ce1046/revoke-keys/0.log" Nov 02 21:47:02.936164 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:02.936130 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e88e0e13-kxptb_abb809b2-9743-4dd3-9dc3-683aae286822/revoke-keys/0.log" Nov 02 21:47:05.102073 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:05.102039 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f74f7c786-fr9rr_44b1f0d7-2146-4520-bfc1-f7c56ddd37be/authorino/0.log" Nov 02 21:47:05.521364 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:05.521335 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2mk96_cbda65e8-653a-4cf7-8e99-fecf2db77fd9/manager/0.log" Nov 02 21:47:06.468944 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:06.468905 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8wmrt_c7428739-8fc4-4d56-9eeb-8018ac0334f3/manager/0.log" Nov 02 21:47:08.441133 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:08.441099 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b579bndl_14f35ad8-9615-4438-9f8d-5f2141131f13/istio-proxy/0.log" Nov 02 21:47:08.718798 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:08.718714 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-h8bs4_d0bf9a44-5ba5-442c-b9c3-443ff0678c9e/discovery/0.log" Nov 02 21:47:09.754202 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:09.754174 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-2sf82_e7d44f90-b9db-4554-9111-75088930d250/istio-proxy/0.log" Nov 02 21:47:10.043355 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:10.043243 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-7h7jm_48b27c1e-b9de-4d19-9b28-656ee6a0f697/istio-proxy/0.log" Nov 02 21:47:10.642868 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:10.642841 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-h2gr5_6eaf89b5-5d8f-4515-9c00-cef33ea31033/istio-proxy/0.log" Nov 02 21:47:13.168832 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:13.168803 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6668f8f7-2vjcr_6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/router/0.log" Nov 02 21:47:14.571631 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:14.571598 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-76pbs_649c1f83-2c79-4258-9822-4ed4628ad39e/main/0.log" Nov 02 21:47:14.837866 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:14.837792 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-6mdnk_f2446135-0668-4948-99a4-cabed80f52cb/main/0.log" Nov 02 21:47:15.371205 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:15.371167 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b842pdq_6f51894d-a930-42e9-ab5a-a1727a3b5c71/main/0.log" Nov 02 21:47:15.916004 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:15.915972 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-xqqjw_8bbd455b-6e9e-4c24-a9dd-3a029dede190/main/0.log" Nov 02 21:47:16.189970 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:16.189889 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-mptr6_16c42059-424f-41a1-9108-ba08f2c79af9/main/0.log" Nov 02 21:47:22.696127 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:47:22.696086 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:47:25.049653 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:25.049613 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz"] Nov 02 21:47:25.094608 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:25.094573 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf6212-mc8nz"] Nov 02 21:47:25.624492 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:25.624447 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84882581-4345-416a-8076-2bc35fa56aaa" path="/var/lib/kubelet/pods/84882581-4345-416a-8076-2bc35fa56aaa/volumes" Nov 02 21:47:27.858902 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:27.858864 5800 scope.go:122] "RemoveContainer" containerID="c645df29ee0ca60ceb4db2dd9167230ed7c512cddbc1bcf8d611a50f57724800" Nov 02 21:47:27.868460 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:27.868434 5800 scope.go:122] "RemoveContainer" containerID="fffc9d5a153ea22a76518f18e93d8106fc1f24c728ff4d8beb5360fd401e7303" Nov 02 21:47:36.044267 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:36.044228 5800 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq"] Nov 02 21:47:36.091015 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:36.090980 5800 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db968f7a-gwltq"] Nov 02 21:47:36.095476 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:36.095448 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j7kp4_eceae17b-5b0b-45ce-87cf-9d720583c247/global-pull-secret-syncer/0.log" Nov 02 21:47:37.301604 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:37.301578 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6s4x6_e5640e81-834c-4854-9cc9-589ccbd39d64/connection-checker/0.log" Nov 02 21:47:37.620483 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:37.620404 5800 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd029bea-c23a-440f-a179-f08627ce1046" path="/var/lib/kubelet/pods/bd029bea-c23a-440f-a179-f08627ce1046/volumes" Nov 02 21:47:37.740831 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:37.740799 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qwntm_cb5fc542-006a-4da5-b985-cc7b0f1709a7/connection-checker/0.log" Nov 02 21:47:38.187346 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:38.187318 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-t25fz_f98defbd-aa11-4a76-bac3-4d7aacdb44a1/connection-checker/0.log" Nov 02 21:47:39.257903 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:39.257874 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-trg79_7c9766ef-0c1a-4ddc-9918-e0c1fb68aada/konnectivity-agent/0.log" Nov 02 21:47:40.636353 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:40.636319 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-13.us-west-2.compute.internal_7161e7f2a8cfc7fa3ac048d9b7921179/haproxy/0.log" Nov 02 21:47:45.770986 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:45.770946 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f74f7c786-fr9rr_44b1f0d7-2146-4520-bfc1-f7c56ddd37be/authorino/0.log" Nov 02 21:47:47.016170 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.016139 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-2mk96_cbda65e8-653a-4cf7-8e99-fecf2db77fd9/manager/0.log" Nov 02 21:47:47.151659 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.151568 5800 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf"] Nov 02 21:47:47.152092 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.152073 5800 cpu_manager.go:414] "containerMap: removing container" podUID="d731c67d-1af9-4e46-b451-7d4de48e5147" containerName="revoke-keys" Nov 02 21:47:47.152200 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.152096 5800 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d731c67d-1af9-4e46-b451-7d4de48e5147" containerName="revoke-keys" Nov 02 21:47:47.152272 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.152257 5800 memory_manager.go:365] "RemoveStaleState removing state" podUID="d731c67d-1af9-4e46-b451-7d4de48e5147" containerName="revoke-keys" Nov 02 21:47:47.156907 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.156887 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.207954 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.207912 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9pw5\"/\"openshift-service-ca.crt\"" Nov 02 21:47:47.254212 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.254179 5800 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k9pw5\"/\"kube-root-ca.crt\"" Nov 02 21:47:47.254212 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.254179 5800 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k9pw5\"/\"default-dockercfg-sn7x5\"" Nov 02 21:47:47.257176 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.257150 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf"] Nov 02 21:47:47.332703 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.332666 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hqs\" (UniqueName: \"kubernetes.io/projected/2a4db29a-5bb9-4281-a3f2-02963b7a6544-kube-api-access-v8hqs\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.332888 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.332774 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-sys\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.332888 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.332870 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-podres\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.333010 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.332894 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-proc\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.333010 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.332938 5800 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-lib-modules\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434443 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434352 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-lib-modules\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434443 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hqs\" (UniqueName: \"kubernetes.io/projected/2a4db29a-5bb9-4281-a3f2-02963b7a6544-kube-api-access-v8hqs\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434492 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-sys\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434502 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-lib-modules\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434578 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-sys\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434589 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-podres\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434628 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434618 5800 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-proc\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434926 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434693 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-podres\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.434926 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.434762 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2a4db29a-5bb9-4281-a3f2-02963b7a6544-proc\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.487203 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.487170 5800 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hqs\" (UniqueName: \"kubernetes.io/projected/2a4db29a-5bb9-4281-a3f2-02963b7a6544-kube-api-access-v8hqs\") pod \"perf-node-gather-daemonset-nrxjf\" (UID: \"2a4db29a-5bb9-4281-a3f2-02963b7a6544\") " pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:47.769662 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:47.769632 5800 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:48.135207 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:48.135168 5800 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf"] Nov 02 21:47:48.136375 ip-10-0-134-13 kubenswrapper[5800]: W1102 21:47:48.136333 5800 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2a4db29a_5bb9_4281_a3f2_02963b7a6544.slice/crio-d91d5c95b35b12ed0b4ae1c1940b9a46b01dc66a5bb660ea5531da9f89987385 WatchSource:0}: Error finding container d91d5c95b35b12ed0b4ae1c1940b9a46b01dc66a5bb660ea5531da9f89987385: Status 404 returned error can't find the container with id d91d5c95b35b12ed0b4ae1c1940b9a46b01dc66a5bb660ea5531da9f89987385 Nov 02 21:47:48.340650 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:48.340607 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8wmrt_c7428739-8fc4-4d56-9eeb-8018ac0334f3/manager/0.log" Nov 02 21:47:49.112085 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:49.112047 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" event={"ID":"2a4db29a-5bb9-4281-a3f2-02963b7a6544","Type":"ContainerStarted","Data":"8937ae468d92d5e4cef590f0b5bb13c7de7902c32d97f62d65009b60d8b6e053"} Nov 02 21:47:49.112299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:49.112112 5800 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" event={"ID":"2a4db29a-5bb9-4281-a3f2-02963b7a6544","Type":"ContainerStarted","Data":"d91d5c95b35b12ed0b4ae1c1940b9a46b01dc66a5bb660ea5531da9f89987385"} Nov 02 21:47:49.112299 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:49.112161 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:49.219828 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:49.219776 5800 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" podStartSLOduration=2.2197603790000002 podStartE2EDuration="2.219760379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:47:49.217841538 +0000 UTC m=+7167.734614125" watchObservedRunningTime="2026-11-02 21:47:49.219760379 +0000 UTC m=+7167.736532980" Nov 02 21:47:52.696657 ip-10-0-134-13 kubenswrapper[5800]: E1102 21:47:52.696608 5800 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6hpx" containerName="extractor" Nov 02 21:47:54.991743 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:54.991706 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-kz9f7_97562254-d922-4415-9438-ae340b218942/cluster-monitoring-operator/0.log" Nov 02 21:47:55.138123 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:55.138082 5800 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k9pw5/perf-node-gather-daemonset-nrxjf" Nov 02 21:47:57.616224 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:57.616190 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xf99_39bb832f-2158-430f-996c-b715e3c262c1/node-exporter/0.log" Nov 02 21:47:58.028442 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:58.028415 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xf99_39bb832f-2158-430f-996c-b715e3c262c1/kube-rbac-proxy/0.log" Nov 02 21:47:58.441265 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:47:58.441191 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9xf99_39bb832f-2158-430f-996c-b715e3c262c1/init-textfile/0.log" Nov 02 21:48:03.228633 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:48:03.228595 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b579bndl_14f35ad8-9615-4438-9f8d-5f2141131f13/istio-proxy/0.log" Nov 02 21:48:08.547528 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:48:08.547491 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-h8bs4_d0bf9a44-5ba5-442c-b9c3-443ff0678c9e/discovery/0.log" Nov 02 21:48:09.293822 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:48:09.293798 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-2sf82_e7d44f90-b9db-4554-9111-75088930d250/istio-proxy/0.log" Nov 02 21:48:09.501127 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:48:09.501091 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-7h7jm_48b27c1e-b9de-4d19-9b28-656ee6a0f697/istio-proxy/0.log" Nov 02 21:48:09.918993 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:48:09.918955 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-h2gr5_6eaf89b5-5d8f-4515-9c00-cef33ea31033/istio-proxy/0.log" Nov 02 21:48:10.812323 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:48:10.812271 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b579bndl_14f35ad8-9615-4438-9f8d-5f2141131f13/istio-proxy/0.log" Nov 02 21:48:11.825082 ip-10-0-134-13 kubenswrapper[5800]: I1102 21:48:11.825041 5800 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f6668f8f7-2vjcr_6f73e2f3-6f98-41bd-abec-b72d60ac2bb5/router/0.log"